JP2000268179A - Three-dimensional shape information obtaining method and device, two-dimensional picture obtaining method and device and record medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method for easily obtaining the highly precise three- dimensional shape information of an object from the dynamic image of the object photographed by a camera to which calibration is not preliminarily applied without using any exclusive special device and a method for obtaining the two-dimensional picture of an object from an arbitrary point of view. SOLUTION: An object is photographed by a camera to which calibration is not preliminarily applied, and the dynamic image is obtained (A), and the optical parameter of the camera and the moving locus of the camera at the time of photographing is obtained based on the dynamic image (B), and the three dimensional shape information of the object is obtained according to a stereoscopic method by using the obtained optical parameter and moving locus and the dynamic image data (C). Then, the two-dimensional picture of the object is obtained from a designated arbitrary point of view by using the obtained optical parameter and moving locus and the dynamic image data and the obtained three-dimension shape information (D).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and an apparatus for acquiring information on a three-dimensional shape of an object, a method and an apparatus for acquiring a two-dimensional image of an object from an arbitrary viewpoint, and to implement those methods. Recording medium on which a program for recording is recorded.

[0002]

2. Description of the Related Art With the development of CAD and CG techniques, it has become necessary to obtain information on the three-dimensional shape of an object and to input the three-dimensional shape information to a computer system. As a method of acquiring information on the three-dimensional shape of an object,
Conventionally, the stereo method using stereo vision (multi-view) is relatively accurate and the algorithm is stable,
It is gradually being introduced into real systems. The stereo method is roughly classified into an active stereo method and a passive stereo method.

The active stereo method is a method of actively irradiating a target object with a laser slit light or the like in order to determine a corresponding point between multi-views, which is the basis of the stereo method.
A high-speed, high-accuracy system can be constructed. However,
Since dedicated hardware is required, there is a problem that it is expensive. In addition, an environment in which the slit light is not hindered by disturbance is usually required, and there is a problem that system suitability is low. Therefore, it has not been widely used outside of application fields where the environment can be restricted and high information accuracy is required.

On the other hand, in the passive stereo method, a plurality of cameras are fixedly arranged, and information on the three-dimensional shape of an object is obtained using only images obtained by the cameras. Therefore, there is an advantage that special hardware other than a general video camera and a still camera is not required and the environment is not restricted, and it has begun to spread widely, and many specific proposals have been made.

[0005] Among them, IEEE TRANSACTIONS ON PAT
TERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.15, N
Muitiple-ba proposed in O.4, APRIL 1993, pp353-363
The seline stereo (multiple baseline stereo) method eliminates the instability in the corresponding point determination process, which has been a problem in the passive stereo method, by using multiple viewpoint images with a simple algorithm. A range image can be obtained. Also, Takeo Kanade, C. Lawrence Zit
Volumetric iterative approach to developed by nick and others
In the stereo matching and occlusion detection method, it is possible to detect an occlusion region and obtain higher-quality three-dimensional shape information in a voxel data format.

[0006] Apart from the method of obtaining the three-dimensional shape information of the object from the photographed image as described above, a two-dimensional image of the object from an arbitrary viewpoint (a virtual camera having an arbitrary position and orientation) designated by a user is obtained. A method of synthesizing and acquiring multiple captured images (Image-based rendering: Image Based Re
ndering (hereinafter abbreviated as IBR)) is known. This two-dimensional image acquisition method focuses on constructing a shape model of an object from three-dimensional shape information, whereas the three-dimensional shape information acquisition method focuses on building a shape model of an object in a computer without building a shape model of the object. The main purpose is to generate a two-dimensional image of the object and present it to the user.
In particular, in systems using virtual reality, training simulations, and the like, it is more important to be able to present an accurate two-dimensional image of an object from an arbitrary viewpoint than to be able to accurately construct a shape model of the object. .

As an example of such an IBR, Steven
A method developed by M. Seitz et al. (Steven M. Seitz, "Toward In
teractive Scene Walkthroughs from Images "). In this method, a voxel space (Voxel Space) containing a target object is considered as a collection of small rectangular parallelepipeds called voxels, and a specific voxel is captured by the imaging surface of each camera. In the case of voxels constituting the surface of an object, the projected images to all cameras have the same color. Therefore, when the projected images to all cameras have the same color, The voxel is assigned the color of the projected image, otherwise no color is assigned to the voxel that is not the surface of the object, and an arbitrarily specified position is determined based on the color of each voxel thus assigned. The two-dimensional image viewed from the virtual camera is synthesized by projecting these onto the imaging surface of the virtual camera having the posture.

After constructing a shape model of the object using the acquired three-dimensional shape information of the object, the shape model is projected on an imaging plane of a virtual camera to obtain a two-dimensional image of the object from an arbitrary viewpoint. A method is conceivable. However, in this method, it is necessary to provide phase information such as a triangular patch based on the acquired three-dimensional shape information, and there is a possibility that inconsistency with an actual object may occur. On the other hand, since the IBR uses a very simple data structure of a voxel space to which colors are assigned, inconsistency with an actual object hardly occurs.

[0009]

By the way, the aforementioned 2
Many passive stereo methods, including one, can provide highly accurate results, but need to calibrate the camera's internal parameters (focal length, etc.) and external parameters (position, orientation, etc.) in advance. . This means that it is not possible to move a plurality of cameras that have been calibrated once, which is a great limitation in actual information acquisition processing. In addition, there is a problem in that equipment for fixing the camera is required, and substantially special and large hardware must be used.

On the other hand, a method of calibrating various parameters of a camera from a moving image of a photographed object and acquiring information on its three-dimensional shape (Marc Pollefeys, Reinhard
Koch and Luc Van Gool, "Self-Calibration and Metr
ic Reconstruction in spiteof Varying and Unknown I
However, this method also requires that the user manually indicate feature points on the image, and obtains information on the three-dimensional shape of the object only by tracking the feature points. Therefore, there is a problem that the accuracy of the shape information is low, and an accurate three-dimensional shape cannot be restored for an object having a complicated shape.

Also, when performing IBR, it is necessary to calibrate the internal parameters and the external parameters of each camera for acquiring a captured image in advance, as in the passive stereo method. Therefore, there is a problem similar to the passive stereo method described above.

[0012] The present invention has been made in view of such circumstances, and does not require calibration in advance and does not use large hardware. Therefore, it is extremely easy to obtain information on the three-dimensional shape of an object. 3D shape information acquisition method and device that can be acquired with high accuracy, 2D image acquisition method and device that can acquire a 2D image of an object from any viewpoint with high accuracy, and their acquisition methods It is an object of the present invention to provide a recording medium on which a program to be executed is recorded.

[0013]

According to a first aspect of the present invention, there is provided a method for obtaining three-dimensional shape information of an object from an image of the object taken by a camera moving relatively to the object. A first step of obtaining an internal parameter of the camera and a relative movement trajectory of the camera at the time of shooting based on the shot image, and a step of calculating the obtained internal parameters and the movement track and the shot image. Using the stereo method to obtain information on the three-dimensional shape of the object.

According to a second aspect of the present invention, in the first aspect, a still object is photographed by a moving camera to obtain a photographed image of the object.

A three-dimensional shape information acquiring method according to a third aspect is characterized in that, in the first aspect, a moving object is photographed with a stationary camera to obtain a photographed image of the object.

According to a third aspect of the present invention, in the method for acquiring three-dimensional shape information according to any one of the first to third aspects, the first step includes a step of extracting a feature point in an arbitrary frame in the photographed image; Tracking the extracted feature points over a plurality of frames in the captured image;
Obtaining the internal parameters and the movement trajectory based on the tracking result.

According to a fifth aspect of the present invention, in the method for obtaining three-dimensional shape information according to the fourth aspect, the moving direction of the feature point is predicted by using the once determined movement trajectory. Are compared, the tracking result of the feature point is corrected based on the comparison result, and the internal parameters and the movement trajectory are obtained again based on the corrected tracking result.

According to a sixth aspect of the present invention, in the method for obtaining three-dimensional shape information according to any one of the first to fifth aspects, a factor decomposition method is used when obtaining the movement locus of the camera based on the tracking result of the feature point. It is characterized by doing.

A three-dimensional shape information acquiring method according to claim 7 is characterized in that in any one of claims 1 to 6, a multiple baseline stereo method is used as the stereo method.

According to a third aspect of the present invention, there is provided an apparatus for obtaining three-dimensional shape information of an object from an image of the object taken by a camera moving relatively to the object. Means for obtaining an internal parameter of the camera based on the image and a relative movement trajectory to the object at the time of shooting by the camera, and a stereo method using the obtained internal parameter and the movement trajectory and the captured image in accordance with a stereo method. Means for obtaining three-dimensional shape information.

According to a ninth aspect of the present invention, there is provided the three-dimensional shape information acquiring apparatus according to the eighth aspect, further comprising the camera which moves relatively to the object.

According to a tenth aspect of the present invention, there is provided a two-dimensional image acquiring method, wherein information obtained by the three-dimensional shape information acquiring method according to any one of the first to seventh aspects is used. A method for acquiring an image, using an internal parameter and a movement trajectory obtained in the first step, the photographed image, and information on a three-dimensional shape obtained in the second step, from an arbitrary designated viewpoint. A two-dimensional image of the object is obtained.

A two-dimensional image acquiring method according to claim 11, wherein a two-dimensional image of the object from an arbitrary viewpoint is acquired based on an image of the object taken by a camera moving relatively to the object. A first step of obtaining an internal parameter of the camera and a relative movement trajectory with respect to the object at the time of shooting by the camera based on the shot image, and using the obtained internal parameters and the movement track and the shot image To obtain three-dimensional shape information of the object by
And a third step of obtaining a two-dimensional image of the object from an arbitrarily designated viewpoint using the obtained internal parameters and the movement trajectory, the captured image, and the obtained three-dimensional shape information. It is characterized by.

A two-dimensional image acquisition method according to a twelfth aspect is characterized in that, in any of the tenth and eleventh aspects, when acquiring a two-dimensional image of the object, image-based rendering is used.

According to a thirteenth aspect of the present invention, there is provided a two-dimensional image acquiring apparatus for acquiring a two-dimensional image of the object from an arbitrary viewpoint based on an image of the object taken by a camera moving relatively to the object. In, means for obtaining a relative trajectory of the internal parameters of the camera and the object at the time of shooting of the camera based on the captured image,
Means for determining the three-dimensional shape information of the object using the determined internal parameters and the movement locus and the captured image, using the determined internal parameters and the movement locus, the captured image and the determined three-dimensional shape information, Means for acquiring a two-dimensional image of the object from an arbitrarily designated viewpoint.

According to a fourteenth aspect of the present invention, there is provided a computer-readable storage medium storing a program for obtaining information on a three-dimensional shape of an object from an image of the object taken by a camera that moves relatively to the object. Program code means for causing the computer to execute an internal parameter of the camera and a relative movement trajectory with respect to the object at the time of shooting by the camera based on the shot image on the readable recording medium; Program code means for causing the computer to execute the acquisition of information on the three-dimensional shape of the object in accordance with a stereo method using the internal parameters and the movement trajectory and the captured image.

A recording medium according to a fifteenth aspect stores a program for acquiring a two-dimensional image of the object from an arbitrary viewpoint based on an image of the object taken by a camera that moves relatively to the object. On the recorded computer-readable recording medium, the computer is caused to execute a process of obtaining an internal parameter of the camera and a relative movement trajectory to the object at the time of shooting by the camera based on the shot image. Program code means, program code means for causing the computer to obtain the three-dimensional shape information of the object using the obtained internal parameters and movement trajectory and the photographed image, and the obtained internal parameters and movement trajectory; Using the photographed image and the obtained three-dimensional shape information, the object is viewed from an arbitrarily designated viewpoint. And having a program code means for executing to obtain the original image on the computer.

FIG. 1 is a diagram showing the concept of a method for acquiring three-dimensional shape information according to the present invention. In the present invention, a target object is photographed by a camera that has not been calibrated in advance, and a moving image is obtained (A). Then, from the moving image, the internal parameters of the camera (such as the focal length) and the movement locus of the camera (such as the position and orientation of the camera at the time of shooting) are obtained (B). Using the obtained information and the moving image, information on the three-dimensional shape of the object is obtained according to the stereo method (C).

According to the present invention, three-dimensional shape information can be obtained from a moving image freely photographed in an arbitrary environment without the need for camera calibration, the degree of freedom in photographing timing is high, and there are no environmental restrictions at all. . Further, equipment for fixing the camera is not required, and the processing can be performed at low cost. Furthermore,
Since three-dimensional shape information is obtained according to the stereo method, highly accurate three-dimensional shape information of the object can be obtained.

In the present invention, the following processing is performed when obtaining the internal parameters and the movement locus of the camera.
First, feature points in an arbitrary one frame in a moving image are extracted, and the extracted feature points are tracked over a plurality of frames in the moving image to obtain a feature point tracking result. Then, based on the feature point tracking result, the internal parameters of the camera are determined (B1). Based on the determined internal parameters and the feature point tracking results, a camera trajectory is obtained using a factor decomposition method (B2). Then, the obtained trajectory is matched with the feature point tracking result, the tracking result data that does not match is excluded, and the internal parameters are determined again (B
1) Based on the re-determined internal parameters and the feature point tracking result excluding the tracking result data that does not match, the movement trajectory of the camera is obtained again by using the factor decomposition method (B).
2). Such a calibration process is repeated several times to obtain final camera internal parameters and trajectories. Therefore, it is possible to obtain various accurate parameters of the camera required in the processing by the stereo method.

In the present invention, a multiple baseline stereo method is used as the stereo method. In the multiple baseline stereo algorithm, the accuracy of the obtained three-dimensional shape information is improved as the number of frames of images having different viewpoints increases. In the present invention, since a moving image captured by a camera is used, a very large number of multi-viewpoint images can be provided for the algorithm of the multiple baseline stereo method, the data density is increased, and the accuracy of the three-dimensional shape information is improved. Can be expected.

FIG. 2 is a diagram showing the concept of a two-dimensional image acquisition method according to the present invention. 2, the same reference numerals are given to the same processing portions as those in FIG. 1, and the description thereof will be omitted. A two-dimensional image of an object from an arbitrary viewpoint using the internal parameters and the movement trajectory of the camera determined in (B), the captured moving image, and the three-dimensional shape information of the object acquired in (C). (D).

According to the present invention, a two-dimensional image of an object from an arbitrary viewpoint can be obtained from a moving image freely photographed in an arbitrary environment without the need for camera calibration. There are no environmental restrictions.
Further, equipment for fixing the camera is not required, and the processing can be performed at low cost.

In the present invention, an IBR is used to acquire a two-dimensional image.
Is used. In this IBR, the smaller the parallax between a captured image and a two-dimensional image from an arbitrary viewpoint, the more accurate a two-dimensional image can be synthesized. In the present invention, a large number of captured images are obtained by a camera that moves relatively to an object, and a high quality image is obtained based on a captured image in which a parallax with a two-dimensional image from an arbitrary viewpoint is small. Can always be obtained.

[0035]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be specifically described below with reference to the drawings showing the embodiments. FIG. 3 is a block diagram showing a configuration of the three-dimensional shape information acquiring device and the two-dimensional image acquiring device of the present invention. In FIG. 3, the main control unit 1 is specifically constituted by a CPU, and is connected to each of the hardware units described below via a bus 2 to control the operation of the hardware units and to perform arithmetic operations described later. Perform various software functions including processing.

The reading section 3 reads analog moving image data of a target object recorded on an external video tape and converts it into digital moving image data. The image memory 4 stores the digital moving image data. RO
M5 stores in advance various software programs necessary for the three-dimensional shape information acquisition processing and the two-dimensional image acquisition processing of the present invention. The RAM 6 is composed of an SRAM, a flash memory, or the like, and stores temporary data generated when executing software. The display unit 7 displays a finally restored three-dimensional shape model of the object, a two-dimensional image of the object from a specified viewpoint, three-dimensional shape information acquisition processing, and intermediate information in the two-dimensional image acquisition processing.

Next, the operation will be described. 4 and 5
5 is a flowchart showing a processing procedure of the three-dimensional shape information acquisition method of the present invention. In the following example, a camera movement trajectory restoring technique using a factorization method is used as a method for obtaining various camera parameters, and a multiple baseline stereo method is used as a stereo method for obtaining three-dimensional shape information. .

(Step S1: Photographing by Video Camera) An object from which three-dimensional shape information is to be obtained is photographed by a video camera. At this time, in order to provide disparity information, it is preferable to take a picture by actively moving the video camera, and how to set the video camera if there is no large blurring in an image in one frame due to rapid camera movement. You may move it. However, subsequent processing cannot be performed on an image in which the camera optical axis and the instantaneous movement direction of the camera are the same. If there is not much parallax between images taken at adjacent positions, a continuous photograph taken by a digital camera or a still camera using ordinary film may be used.

(Step S2: Capture of Moving Image) The reading unit 3 reads analog moving image data of an object photographed by a video camera from an image medium such as a video tape, and digitizes the analog moving image data. Digital moving image data is obtained. Instead of reading out moving image data from an image medium, moving image data may be directly taken in from a video camera in real time. The obtained digital moving image data is temporarily stored in the image memory 4.

(Step S3: Preprocessing of Moving Image) In order to smoothly perform a feature point extraction process and a feature point tracking process described later, some preprocessing is performed on a digital moving image.

[Image Interpolation Processing] In the video video signal standard NTSC used in Japan and the United States, interlaced scanning is performed, so that an image for one field has a comb-like shape. In the case of a moving image having a large movement, the image may be greatly different in the horizontal direction due to a difference in scanning time between the odd field and the even field. This phenomenon causes a large error in feature point extraction and feature point tracking using the differential amount of luminance in the image plane. Therefore, in this example, only one of the odd field and the even field is used, and the other field image is created by vertical interpolation processing.

[Pyramid Image and Optical Flow Calculation] An optical flow is calculated in advance in order to link a point feature detected in each frame with a feature point which is considered to be the same point on the preceding and succeeding frames. Since a moving image is an image captured continuously in time, it can be differentiated with respect to time. Brightness I of the same point in three-dimensional space
Is not changed between adjacent frames, a spatio-temporal differential equation such as the following equation (1) is established between two consecutive frames in a moving image. In the equation (1), Ix, I
y and It are partial derivatives of the image in the horizontal direction, the vertical direction, and the time direction, respectively.
It is called optical flow.

[0043]

(Equation 1)

In order to solve the spatio-temporal differential equation, in this example, the relaxation method is used. Since this method is basically an iterative asymptotic method, the initial value must be close to the solution to some extent. Therefore, in this example, the image is １ ／, １ ／
This method is applied after reducing the distance to the corresponding point by reducing the size to such a size. Thereafter, the optical flow of the next large image is calculated using the optical flow of the small image as an initial value. This optical flow is calculated over all frames for generating feature point tracking data.

(Step S4: Feature Point Extraction) Feature points are extracted from each frame. A feature point such as a vertex of an object in the three-dimensional space is detected as a position where the differential value of the luminance in the image is largely changed irrespective of its direction.

(Step S5: Feature Point Tracking) Step S
The feature points extracted in each frame are connected using the optical flow calculated in step 3. The position where a certain pixel on an arbitrary frame moves in the next frame is predicted from the optical flow, and the feature points between the two frames are connected as the same point in the three-dimensional space. At this time, the idea described in step S4 is also used.

(Step S6: Camera Internal Parameter Self-Calibration) The internal parameters of the camera used for photographing are calibrated from the feature point tracking data obtained up to step S5. Internal parameters of the camera include a lens focal length, image center coordinates, a pixel aspect ratio, and the like. Of these, the normal lens is known except for the lens focal length, so it is common to calibrate substantially only the lens focal length. However, calibration of the image center coordinates and the pixel aspect ratio may be required. This step S
The algorithm illustrated in FIG. 6 can determine all these internal parameters.

It is known that the projection matrix of an arbitrary initial frame in a moving image can be expressed as P ₁ = [I | 0]. The projection matrix in other frames can be represented by using a fundamental matrix between the initial frame and this frame. The projection matrix P 'of the frame of interest can be expressed by the following equation (3). Here, the matrix M is represented by the following equation (4), and the matrices e and F are the epipole and the fundamental matrix of the frame, respectively. The matrix [] x is defined as shown in the following equation (5) for an arbitrary 3 × 1 vector.

[0049]

(Equation 2)

The projection matrix obtained in this manner, the absolute conic curve Ω on the plane at infinity, and the conic curve ω on the imaging surface, which is the projection thereof, have the relationship expressed by the following equation (6). I know there is. Also,
It is known that the matrices Ω and ω are symmetric matrices as shown in equations (7) and (8), respectively.

[0051]

(Equation 3)

From these results, ω ₁₁ = ω ₂₂ , ω ₁₂ = ω ₁₃ =
Equations such as ω ₂₃ = 0 are derived, but the matrix Ω
Can be obtained by applying a conditional linear least squares method using the projection matrix of each frame. In this case, the rank of the matrix Ω is known to be 3, so once the singular value decomposition is performed,
The matrix is reconfigured with the smallest eigenvalue set to zero. Thereby, the focal length of the first frame is obtained, and the focal lengths of the other frames are obtained by substituting the obtained matrix Ω into Expression (6).

(Step S7: Using Weak Center Projection Model)
Restoration of camera trajectory using factorization method)
From camera parameters and feature point tracking data,
Restore traces. Here, Carnegie Mellon University, USA
The factorization method developed by Professor Takeo Kanade and others is used. factor
The feature of the decomposition method is that the camera moves and the object moves smoothly.
It does not require any constraints such as
To use a singular value decomposition that is computationally stable
stable. On the image of frame j (j = 1,..., M)
(I = 1,..., N) at the coordinates (x_ij, Y_ij)
The coordinate value of the coordinate system with the center of gravity of the point as the origin. Weak central shooting
In the shadow model, feature point coordinates (x_ij, Y _ij) And three
Feature point coordinates (X_i, Y_i) Has the following formula
There is such a relationship as (9).

[0054]

(Equation 4)

Here, s _j is the image scale of the j-th frame relative to the scale of the object, and the vectors r _1j and r _2j
Are the first and second row vectors of the rotation matrix of the camera coordinate system of the j-th frame relative to the object coordinate system, respectively. The scale of the object is the same as the image of the first frame (s ₁ = 1), and the posture of the object in the coordinate system is the same as the camera coordinate system in the first frame (r ₁₁
= [1,0,0] ^T , r ₂₁ = [0,1,0] ^T ).

When all the feature points of all the frames are arranged in one measurement matrix, the following equation (10) is obtained.

[0057]

(Equation 5)

Since both the matrix M indicating the camera movement locus and the matrix S indicating the shape have a rank of 3, the rank of the matrix D must be 3. However, the fourth and subsequent singular values do not become zero due to the failure of the feature point tracking and the quantization error. Therefore, when the singular value is decomposed, the following equation (11) is obtained. Assuming these as matrices V ', Σ', and U ', a matrix D' obtained by reconstructing the matrix D is obtained as equation (12).

[0059]

(Equation 6)

This matrix D 'is a rank 3 matrix that minimizes the following (13). Although it is desired to obtain the matrices M and S from the matrix D ', the combination of the matrices M and S is not unique. This is because an arbitrary 3 × 3 regular matrix C satisfies the following equation (14).

[0061]

(Equation 7)

Therefore, a matrix C satisfying the following equation (15) is obtained. It is known that the following equation (16) is obtained from the constraint condition of the weak center projection.

[0063]

(Equation 8)

From the equation (16), considering a matrix B satisfying the following equation (17), (2m + 1) linear equations relating to the six components of the matrix B are obtained. The components of the matrix B are determined by the least square method from these equations, and further, r _1j = [1,0,0] ^T , r _2j = [0,
The matrix C can be obtained by adding the condition of [1,0] ^T. From this matrix C, the camera movement locus is determined as in the following equation (18).

[0065]

(Equation 9)

Here, it is possible to obtain the shape from the following equation (19). However, since this shape is obtained only from the tracking results of the feature points and its accuracy is low, Not used in the example.

[0067]

(Equation 10)

(Steps S8 and S9: Evaluation of Tracking of Feature Point) The tracking result of the feature point obtained in step S5 is evaluated for an error using the restored camera movement trajectory, and tracking data considered to have failed in tracking is obtained. Remove.

Using the internal parameters of the camera obtained in step S6 and the camera trajectory obtained in step S7, a fundamental matrix between each frame image is obtained. This fundamental matrix F is specifically determined by the following equation (20). Here, the matrix A represents an internal parameter of the camera, and is specifically expressed by the following equation (21). However, the pixel center is (0, 0) and the aspect ratio is 1: 1. Matrix E
Is called an essential matrix, and is specifically expressed by the following equation (22). Note that the matrix R represents rotation during camera movement.

[0070]

[Equation 11]

From the fundamental matrix obtained in this way, epipolar lines on adjacent frames corresponding to feature points can be obtained. The feature point we are paying attention to should be on this epipolar line in adjacent frames. Therefore, a distance between a feature point position in an adjacent frame and an epipolar line on which the feature point should be placed is used as an evaluation criterion of tracking accuracy. It is specifically expressed by the following equation (23).

[0072]

(Equation 12)

In equation (23), the accuracy is better as ω is larger. When the value of ω exceeds a predetermined threshold, the tracking failure is determined and the data of the feature point is deleted from the camera movement tracking data. In this algorithm, since the calculation of the matrix F itself includes the data of the feature point whose tracking has failed, it is necessary to return to step S6 to obtain a more correct matrix F. By repeating such a cycle a plurality of times, it is possible to obtain data having an accuracy that can withstand the application of the algorithm of the subsequent multiple baseline stereo method.

(Step S10: Restoration of Camera Movement Locus Using Nonlinear Least Square Method Based on Perspective Model) In the weak center projection model used in step S7, image coordinates and three-dimensional coordinates are represented by linear functions. However, the central projection model is reasonable for a real camera,
It has a nonlinear relationship as shown in the following equation (24). Note that f
Is the focal length.

[0075]

(Equation 13)

Therefore, an evaluation function such as the following equation (25) is set.

[0077]

[Equation 14]

This equation (25) is the sum of squares of the distance error between the image of the three-dimensional object projected through the three-dimensional motion and the actually obtained image. Using the camera movement and the shape restoration result obtained in step S7 as initial values, Levenberg-Marguard
Solve this nonlinear least squares problem using the t method. As a result, a more accurate camera movement trajectory is restored.

(Step S11: Determination of Base Image) A base image to be used for the subsequent multiple baseline stereo method is determined based on the restored camera movement trajectory and the coordinate values of the feature points. In the multiple baseline stereo method, an image (frame) for which a distance image is to be generated is called a base image. A plurality of images (frames) referred to for generating the distance image are referred to as reference images.
When shooting with a video camera, a distance image can be obtained regardless of the plurality of frames as a base image, but depending on the shape of an object, an accurate three-dimensional model may not be expected due to selection of the base image.

FIG. 6 is a diagram for explaining the reason. In the example shown in FIG. 6A, since the angle α between the surface of the target object 10 and the optical axis of the video camera 11 is greatly deviated from a right angle, both the data density and the accuracy are deteriorated.
On the other hand, in the example shown in FIG. 6B, since the angle α is close to a right angle, both the data density and the accuracy are improved.

Accordingly, Delaunay mesh generation is performed on the feature points extracted in step S4. The coordinate values of the feature points in the three-dimensional space determined in step S10 are given to this mesh, and a rough three-dimensional shape model of the object is obtained. From the three-dimensional shape model and the camera movement trajectory, it is evaluated how suitable each frame is for generating a range image of the object. The specific evaluation value at this time is given by the following equation (26).
Give in. Note that s _ij is the area of the image in the frame of interest for the i-th triangular patch visible in the j-th frame.
The vector n _ij is a representative normal vector of the i-th triangular patch seen in the j-th frame. The vector c _j is a unit vector representing the camera z-axis of the j-th frame.

[0082]

(Equation 15)

The evaluation value represented by the equation (26) indicates the visibility of the object plane in the frame of interest. The base image candidate is determined based on this evaluation value. However, since the base image candidate may be biased only to a frame photographed from a specific direction as it is, the camera movement trajectory is appropriately divided at a predetermined distance, and within each camera movement trajectory segment. The above processing is performed, and a frame having a high evaluation value is set as a base frame.

(Step S12: Determination of Range in Zeta Direction)
In the subsequent multiple baseline stereo processing, S
A profile called SSD (Sum of Sum of Squared Difference) for evaluating the correlation of each part between the base image and the reference image is created. An SSSD profile is created along the depth for each pixel of the base image, but this is a very computationally expensive process, and the overall depth depends on the depth region to be calculated (distance from the center of the camera to the object surface). The processing time is almost determined.

However, it is not known how far the object surface is from the center of the camera until normal shape restoration is completed. Therefore, strictly speaking, an SSSD profile must be created to a depth at which parallax used for stereo vision can be detected. No. However, it is not generally conceivable that a distance image that is so far is required, and the time required for creating an SSSD profile is too long to be realistic.

Therefore, based on the coordinate values of the feature points of the object obtained in step S10, the area for creating the SSSD profile is determined. The closest feature point and the farthest feature point are used from the frame, and an area between both points is defined as an SSSD profile creation area. Thus, shape restoration by the multiple baseline stereo method can be achieved in a relatively short time.

(Steps S13 and S14: Shape Restoration by Multiple Baseline Stereo Method) Internal parameters of the camera obtained in step S6, accurate camera movement trajectory obtained in step S10, and base image determined in step S11 And a plurality of reference frames before and after that, an algorithm of a multiple baseline stereo method is executed to generate a dense range image of each base frame.

In stereo vision, a specific point on the base image and a corresponding point on the reference image are searched, parallax is determined from these two points, and depth is identified using the distance between camera positions. . The determination of the corresponding point and the identification of the depth are performed as follows.

A mask having a size of about 10 × 10 pixels is provided on the base image and the reference image, and the sum of squares of the difference in luminance between pixels in these two masks is used as an SSD (sum of s).
quared difference). An epipolar line on the reference image corresponding to the center position of the mask on the base image is obtained from the matrix F given in advance, and the SSD is calculated while moving the mask on the reference image along this. Thus, an SSD profile along the epi-polar is generated. Since the position on the epi-polar corresponds to the depth, this can be said to be an SSD profile for the depth. The position on the epipolar line where the SSD profile shows the minimum value is the corresponding point, and can be estimated to be the depth.

However, when assuming a scene in which, for example, two camera positions are in a horizontal positional relationship and a number of vertical bars are arranged at equal intervals, the SSD profile can have many similar local Since it has the minimum value, the depth cannot be identified. Therefore, the same scene is processed so that three or more camera positions have a horizontal positional relationship. Here, an SSSD defined by the following equation (27) is considered. FIG. 7 is a diagram illustrating the relationship between the parameters in equation (27).

[0091]

(Equation 16)

Here, vector m: N vector representing the pixel of interest on the base image Vector m ': pixel included in the mask set around the pixel of interest on the base image α: from the pixel of interest on the base image N []: Normalization operator dΩ: Small solid angle f _i (): Luminance function Matrix R _i : Rotation matrix of i-frame camera Vector B _i : Translation frame of i-frame camera

Equation (27) means that SSDs are added for a plurality of reference images. SS with depth on the horizontal axis
When D is graphed, each SSD has multiple local minima. SSDs obtained from any reference image have a minimum at some depth of the real surface. Other local minimum values are shifted depending on the position of the camera.
In an SSD, the data is averaged out, but a clear valley is formed at a correct minimum value. In this way, only one distinct minimum is stably detected and depth is identified.
This process is performed on all the pixels of the base image, and a dense range image can be stably obtained.

Since the generation of the SSSD is very expensive, if the SSSD is calculated for a large number of depths even within the SSSD calculation range obtained in step S12, the entire processing time becomes enormous. Therefore, the SSSD is calculated for a depth of about 30, a spline curve passing through the SSSD is obtained, and a smooth SSSD profile is determined by interpolation. The overall minimum value is analytically detected from the determined profile to identify the depth.

(Step S15: Triangular patch model generation)
In the base image determined in step S11, a triangular patch model is generated from the distance image restored by the multiple baseline stereo processing in step S13.

A range image is basically only a group of points having three-dimensional coordinate values. Therefore, three triangular points considered to be adjacent in the three-dimensional space are grouped together to generate a small triangular polygon (patch), and a complex object plane is formed by the set of these triangular polygons, and displayed on the display unit 7. I do. As a simple method, utilizing the fact that the adjacency of the pixels in one distance image substantially holds the adjacency in the three-dimensional space, for example, the pixel of interest and the pixels on the left and below the target pixel Is used to generate a small triangular polygon, and the same processing is applied to all pixels in the distance image.

This method has an advantage that a triangular batch model can be generated stably and at high speed. However, pixels in a distance image generated by a plurality of base images (viewpoints) do not have an appropriate adjacent relationship. Therefore, a method of associating pixels in adjacent distance images using epipolar constraint or the like can be considered, but the processing becomes complicated, and various cases are required. Therefore, even if no adjacency relation of the points is given at all, a marching cube algorithm for generating a triangular polygon by applying appropriate three points from only the coordinate values is applied.

First, the triangular batch generation algorithm using the above-described two-dimensional image adjacency is applied to all distance images. Next, the minimum value and the maximum value in the direction of each coordinate axis of the space occupied by the data are obtained from all the distance images. A voxel space is set based on these values. “1” is arranged at the center of the cell including the three-dimensional coordinate value on the distance image, and “0” is arranged at other positions. A cube is created by connecting the centers of eight adjacent cells. This is called a marching cube. At the eight vertices of this marching cube,
Since “0” or “1” is arranged, there are 2 ⁸ = 256 kinds of arrangement patterns. However, if things that are geometrically symmetric (rotational type, reflective type, inverted type of "0" and "1", etc.) are considered to be the same, it can be finally reduced to 15 types of arrangements . By defining the shape of the triangular polygon to be generated for each of these 15 types of patterns, and applying that pattern to all cubes,
A triangular polygon can be generated as a whole.

Next, an operation of acquiring a two-dimensional image of an object from an arbitrary viewpoint designated by the user (a virtual camera having an arbitrary designated position and orientation) will be described. 8 and 9 are flowcharts showing the processing procedure of the two-dimensional image acquisition method of the present invention. In the following example,
As a technique for acquiring a two-dimensional image of an object, IBR (Image Based Rendering) is used.

(Steps S1 to S14) Since the processing of steps S1 to S14 is the same as the processing procedure of the above-described three-dimensional shape information acquisition, the respective steps are denoted by the same step numbers and will be described. Omitted.

(Steps S21 and S22: Acquisition of Two-Dimensional Image of Object) IBR is executed using various camera parameters, photographed images and three-dimensional shape information of the object obtained in steps S1 to S14. This IBR processing is 2
Divided into two steps.

As a first step, a voxel space including an object is constructed, and a color is assigned to each voxel in the voxel space. This voxel space is determined based on all range images obtained by the multiple baseline stereo method described above.

More specifically, a rectangular parallelepiped region is obtained by using all the distance images and the maximum and minimum values of the x, y, and z values of all three-dimensional coordinates calculated from the position of the camera that measured them. Is set as a voxel space. Next, the voxel space is divided at appropriate intervals to generate each voxel. Next, a point group calculated from the distance image generated by the multiple baseline stereo method is generated in the voxel space.

The technique developed by Steven M. Seitz et al. (“Towa
In rd Interactive Scene Walkthroughs from Images "), processing is performed on all voxels in the voxel space to select colors to be assigned. However, this method takes an enormous amount of time. Therefore, in this example, multiple baselines are used. Processing time can be reduced by using a distance image generated by a stereo method.

For a certain three-dimensional point, a plane perpendicular to the straight line connecting this point and the center of the camera is set, and the size of the pixel of the range image including this point is reflected on this plane. Specifically, a straight line connecting the camera center and the four corners of the rectangular pixel
Intersect with the set plane and generate a rectangle on the plane.
The voxel including this rectangle is flagged as having a surface. FIG. 10 is a diagram illustrating this concept.

Next, the following processing is performed only on the voxel for which this flag is set. A straight line connecting the voxel of interest and the center of each camera is scanned to confirm that the voxel does not pass through another voxel with a flag.
If occlusion must have occurred when passing through another voxel with the flag set, the image of the camera is excluded from the processing for determining the color of the voxel. The color of the pixel at the intersection of the imaging plane when occlusion has not occurred is a color candidate to be assigned to the voxel. Each voxel is assigned an average value of the color candidates of the plurality of pixels.

It is conceivable that there are pixels in the range image for which the processing has failed in the multiple baseline stereo method. In this case, for the evaluation function E defined by the following equation (28) using the standard deviation of the color candidates, an appropriate threshold value is provided, and if the evaluation function E falls below the threshold value, no color is assigned and the flag is also deleted. In equation (28), s is the standard deviation of the color candidate, σ is the standard deviation of the luminance value of the camera imaging device, and m−
1 represents the degree of freedom, these colors candidate and the luminance value is to follow a chi ² distribution.

[0108]

[Equation 17]

In the second step, a two-dimensional image obtained by a virtual camera at an arbitrary position and orientation is generated using the voxel space to which a color is assigned to each voxel as described above. At this time, a technique of performing a simple perspective transformation of a voxel space to which a color is assigned to a virtual camera imaging plane is adopted.
That is, the position on the virtual camera imaging plane is calculated according to the following equation (29). In equation (29), x and y are two-dimensional coordinate values on the imaging surface, f is a focal length, X, Y, and Z are three-dimensional coordinate values at the center of the voxel, and a matrix Tr is a world coordinate system converted to a camera coordinate system. Is a 4 × 4 matrix.

[0110]

(Equation 18)

Even if the case of occlusion is excluded, two or more voxels may be associated with one pixel on the imaging surface. In such a case, the average value is used.

Next, a specific example of the present invention will be described.
FIG. 11 is a schematic diagram showing a shooting state of the target object 10 by the video camera 11, and the object 10 in this example has a shape in which a part of a sphere is attached to one surface of a cube. While moving and rotating the video camera 11, such a stationary object 10 is photographed from various angles to obtain a moving image.

FIG. 12 is a diagram showing feature points (□) extracted in accordance with step S4 described above. The feature points are likely to appear at the vertices of the object 10, but randomly appear on a smooth curved surface according to the surface pattern or the like.

FIG. 13 schematically shows feature point tracking according to step S5 described above, and shows the result of tracking the movement of feature points between two frames (from □ to ○). I have.

FIG. 14 shows an object generated according to the above-described step S15 based on the distance image restored by the multiple baseline stereo method according to the above-described step S13.
It is a figure showing ten triangular patch models. Also, FIG.
FIG. 7 is a diagram illustrating a three-dimensional model restored by a factorization method using only tracking results of feature points, which is a comparative example of the present invention. In this comparative example, the cube portion is accurately restored, but the sphere portion is restored as a coarse polyhedron, and the restoration accuracy is low. On the other hand, in the example of the present invention, the object 10 is represented by fine triangular patches, so that not only the cubic portion but also the sphere portion can be accurately restored. Such a method of constructing a three-dimensional model is a method suitable for measuring the three-dimensional shape itself of an object.

Further, according to the present invention, a two-dimensional image can be obtained from an arbitrary viewpoint by using the camera parameters, the photographed image, and the three-dimensional shape information of the object in accordance with step S21 described above. Such a method of acquiring a two-dimensional image is a method suitable for a virtual reality system, a training simulation, or the like, whose main purpose is to display an object on a display or the like.

FIG. 16 is a diagram showing a configuration example of the three-dimensional shape information acquiring device, the two-dimensional image acquiring device, and the recording medium of the present invention. The program exemplified here includes steps S2 to S15 shown in FIGS. 4 and 5 or steps S2 to S14, S21 and S22 shown in FIGS. 8 and 9 and is recorded on a recording medium described below. .

In FIG. 16, a recording medium 21 that is connected online to the computer 20 is, for example, a WWW (World Wide Web) server computer that is installed separately from a place where the computer 20 is installed. The program 21a is recorded as follows. A computer 20 functioning as a three-dimensional shape information acquiring device and a two-dimensional image acquiring device, a program read from a recording medium 21;
Under the control of 21a, information on the three-dimensional shape of the object is obtained, or a two-dimensional image of the object from any specified viewpoint is obtained.

The recording medium 22 provided inside the computer 20 uses, for example, a hard disk drive or a ROM installed therein. The recording medium 22 stores the program 22a as described above. The computer 20 functioning as the three-dimensional shape information acquiring device and the two-dimensional image acquiring device includes the program 22 read from the recording medium 22.
Under the control of a, information on the three-dimensional shape of the object is obtained, or a two-dimensional image of the object from a specified arbitrary viewpoint is obtained. An example of the recording medium 22 is the ROM 5 in FIG.
Corresponds to.

The recording medium 23 used by being loaded into the disk drive 20a provided in the computer 20 is, for example, a transportable magneto-optical disk, CD-ROM or flexible disk. The program 23a is recorded as follows. The computer 20 functioning as the three-dimensional shape information acquiring device and the two-dimensional image acquiring device has the program 23 read from the recording medium 23.
Under the control of a, information on the three-dimensional shape of the object is obtained, or a two-dimensional image of the object from a specified arbitrary viewpoint is obtained.

In the above example, a moving object is captured by photographing a stationary object with a moving camera. On the contrary, the moving object is stopped and photographed. The moving image of the object may be obtained by shooting with a camera. Further, a moving image of the object may be obtained in a situation where both the object and the camera are moving.

FIG. 17 is a diagram showing another embodiment of the present invention in which a moving object is photographed by a stationary camera. As shown in FIG. 17, a standard video camera 11
Is fixed to a tripod 12, the target object 10 is placed on the turntable 13, and the turntable 13 is rotated. In addition,
For example, a white cloth 14 is provided behind the object 10 so that the background image is constant even when the object 10 moves. Then, the three-dimensional shape of the object 10 is restored from the moving image captured by the video camera 11 in which the rotating object 10 is fixed according to the same method as described above. In such a system, a video camera
11 internal parameters, external parameters, turntable
There is no need for information such as 13 rotation angles. The rotation of the turntable 13 causes the video camera 11 to move relative to the object 10.
Is calculated as the movement trajectory.

In the above example, the multiple baseline stereo method was used when obtaining the three-dimensional shape model of the object, but the volumetric iterative approach to stere
Of course, other methods such as the matching and occlusion detection method may be used.

[0124]

As described above, according to the present invention, various parameters (internal parameters and moving trajectories) of a camera are obtained from a moving image obtained by photographing an object with a camera which has not been calibrated in advance. Since the information of the three-dimensional shape of the object is acquired according to the stereo method using and, the calibration of the camera is unnecessary, and a device for fixing the camera is also unnecessary,
Very easily obtain three-dimensional shape information of the object,
Moreover, it is possible to obtain three-dimensional shape information with extremely high accuracy of the restored shape of the object.

In the present invention, various parameters (internal parameters and movement trajectories) of the camera are obtained from a moving image obtained by photographing the object with a camera which has not been calibrated beforehand, and three-dimensional shape information of the object is obtained. Since the two-dimensional image of the object from an arbitrary viewpoint is obtained using the parameters and the three-dimensional shape information and the moving image, the calibration of the camera is unnecessary, and the device for fixing the camera is also unnecessary. Yes, it is very easy to present a high quality two-dimensional image of an object from any viewpoint.

[Brief description of the drawings]

FIG. 1 is a diagram showing the concept of a method for acquiring three-dimensional shape information according to the present invention.

FIG. 2 is a diagram illustrating the concept of a two-dimensional image acquisition method according to the present invention.

FIG. 3 is a block diagram showing a configuration of a three-dimensional shape information acquiring device and a two-dimensional image acquiring device according to the present invention.

FIG. 4 is a flowchart showing a processing procedure of the three-dimensional shape information acquiring method of the present invention.

FIG. 5 is a flowchart showing a processing procedure of the three-dimensional shape information acquiring method of the present invention.

FIG. 6 is a diagram illustrating a positional relationship between a target object and an optical axis of a camera.

FIG. 7 is a diagram showing various parameters in an SSSD calculation formula.

FIG. 8 is a flowchart showing a processing procedure of the two-dimensional image acquisition method of the present invention.

FIG. 9 is a flowchart showing a processing procedure of the two-dimensional image acquisition method of the present invention.

FIG. 10 is a diagram showing a relationship between a distance image and a voxel space in IBR.

FIG. 11 is a schematic diagram showing a state where a target object is photographed by a video camera.

FIG. 12 is a diagram showing an example of a pattern of extracted feature points.

FIG. 13 is a diagram illustrating an example of a feature point tracking pattern.

FIG. 14 is a diagram showing a three-dimensional model of an object restored according to the present invention.

FIG. 15 is a diagram illustrating a three-dimensional model of an object restored according to a comparative example.

FIG. 16 is a diagram illustrating a configuration example of a three-dimensional shape information acquisition device, a two-dimensional image acquisition device, and a recording medium.

FIG. 17 is a diagram showing another embodiment of the present invention.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Main control part (CPU) 4 Image memory 5 ROM 6 RAM 10 Object 11 Video camera 20 Computer 21, 22, 23 Recording medium

 ────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Katsuya Yasuda 4-25- 9-208 Minami Yata Park, Higashi Sumiyoshi-ku, Osaka-shi, Osaka

Claims (15)

[Claims] 1. A method for obtaining information on a three-dimensional shape of an object from an image of the object taken by a camera that moves relatively to the object, wherein an internal parameter of the camera and the A first step of obtaining a relative movement trajectory with respect to the object at the time of shooting by a camera; and a second step of obtaining information on a three-dimensional shape of the object according to a stereo method using the obtained internal parameters, the movement trajectory, and the captured image. And a method for acquiring three-dimensional shape information. 2. The three-dimensional shape information acquiring method according to claim 1, wherein a still object is photographed by a moving camera to obtain a photographed image of the object. 3. The method according to claim 1, wherein a moving object is photographed by a stationary camera to obtain a photographed image of the object. 4. The first step includes: extracting a feature point in an arbitrary frame in the photographed image; tracking the extracted feature point over a plurality of frames in the photographed image; 4. The method for acquiring three-dimensional shape information according to claim 1, further comprising: obtaining the internal parameters and the movement trajectory based on a result. 5. A moving direction of the feature point is predicted by using a moving trajectory obtained once, a result of the prediction is compared with a result of tracking the feature point, and a result of the tracking of the feature point is determined based on the comparison result. The three-dimensional shape information acquiring method according to claim 4, wherein the internal parameters and the movement trajectory are obtained again based on the corrected tracking result. 6. The method for acquiring three-dimensional shape information according to claim 1, wherein a factor decomposition method is used when obtaining the movement locus of the camera based on the tracking result of the feature points. 7. The three-dimensional shape information acquiring method according to claim 1, wherein a multiple baseline stereo method is used as the stereo method. 8. An apparatus for obtaining information on a three-dimensional shape of the object from an image of the object taken by a camera relatively moving with respect to the object, wherein an internal parameter of the camera and the Means for obtaining a relative movement trajectory with respect to the object at the time of shooting by a camera, and means for obtaining information on the three-dimensional shape of the object according to a stereo method using the obtained internal parameters and the movement trajectory and the captured image A three-dimensional shape information acquisition device, characterized in that: 9. The three-dimensional shape information acquiring apparatus according to claim 8, further comprising the camera that moves relatively to the object. 10. A method for acquiring a two-dimensional image of the object from an arbitrary viewpoint, using information obtained by the method for acquiring three-dimensional shape information according to claim 1. A two-dimensional image of the object is obtained from an arbitrarily designated viewpoint using the internal parameters and the movement trajectory obtained in the first step, the photographed image, and the information of the three-dimensional shape obtained in the second step. A method for acquiring a two-dimensional image, the method comprising: 11. A method for acquiring a two-dimensional image of the object from an arbitrary viewpoint based on an image of the object taken by a camera that moves relatively to the object, wherein the two-dimensional image A first step of obtaining an internal parameter of the camera and a relative trajectory of the camera relative to the object at the time of shooting, and obtaining three-dimensional shape information of the object using the obtained internal parameter and the trajectory and the captured image A second step, and a third step of obtaining a two-dimensional image of the object from an arbitrarily designated viewpoint using the obtained internal parameters and movement trajectory, the captured image, and the obtained three-dimensional shape information. A method for acquiring a two-dimensional image, comprising: 12. The method according to claim 1, wherein when acquiring a two-dimensional image of the object, image-based rendering is used.
12. The two-dimensional image acquisition method according to any one of 0 and 11. 13. An apparatus for acquiring a two-dimensional image of the object from an arbitrary viewpoint based on an image of the object taken by a camera relatively moving with respect to the object, Means for obtaining an internal parameter of the camera and a relative movement trajectory to the object at the time of shooting by the camera; means for obtaining three-dimensional shape information of the object using the obtained internal parameter and movement trajectory and the captured image; Means for acquiring a two-dimensional image of the object from an arbitrarily designated viewpoint using the determined internal parameters and the movement trajectory, the captured image and the determined three-dimensional shape information. Two-dimensional image acquisition device. 14. A computer-readable recording medium storing a program for obtaining information on a three-dimensional shape of an object from an image of the object taken by a camera that moves relatively to the object. Program code means for causing the computer to determine an internal parameter of the camera based on a captured image and a relative movement trajectory to the object at the time of imaging by the camera; and the determined internal parameter and movement trajectory. And a program code unit for causing the computer to obtain three-dimensional shape information of the object according to a stereo method using the captured image. 15. A computer which records a program for acquiring a two-dimensional image of the object from an arbitrary viewpoint based on an image of the object taken by a camera moving relatively to the object. Program code means for causing the computer to execute an internal parameter of the camera and a relative movement trajectory with respect to the object at the time of shooting by the camera based on the shot image on the readable recording medium; and Program code means for causing the computer to obtain three-dimensional shape information of the object using the internal parameters and the movement trajectory and the photographed image; and the obtained internal parameters and movement trajectory and the photographed image and the obtained three-dimensional image. Using the shape information, to obtain a two-dimensional image of the object from an arbitrarily designated viewpoint Recording medium having program code means to be executed by the computer.