JPH11150741A - Three-dimensional picture displaying method and its device by stereo photographing - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reproduce the three-dimensional picture of an object with reality on a monitor screen with the resolution of the same degree as that of a photographed two-dimensional picture by sticking a pixel corresponding to a stereo picture onto each of plural curved surfaces by the unit of the pixel. SOLUTION: Picture data of a stereo photograph is obtained from a stereo digital camera to execute the distortion correction processing of each picture data D1 and D2 and to obtain a distance of each point of a photographed object from the view point origin by a correlation method based on picture data D1 and D2 . Then, coordinate data of each pixel on a boundary of the picture D1 being a main picture is obtained and distance data of each point of the photographed object is edited corrected. Continually, an equation for generating such a curved surface as to interpolate each area for each area is decided based on a point distance data group and boundary data in each area on the same surface. Then, each pixel of the main picture D1 is stuck onto a corresponding interpolation curved surface based on the equation and boundary data to prepare stereoscopic picture data of the photographed object.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for displaying a three-dimensional image by stereo photography and an apparatus using the method.

[0002]

2. Description of the Related Art Conventionally, a subject to be photographed (subject)
As a display method for displaying the image on a monitor screen such as a CRT, a photographing target is first photographed (imaged) from each of a plurality of different positions around the image, and then an image closest to a desired observation direction (angle) is displayed on the monitor screen. An image display method to be displayed on the top is known.

In such a conventional image display method, since a desired observation direction depends on the direction at the time of photographing, an image viewed from an arbitrary viewpoint cannot be reproduced. FIG. 15 shows an example of a shooting situation by the conventional image display method. In this example, the same camera (digital camera) C captures a substantially cuboid-shaped imaging target s from each of a plurality of different positions (four locations) around the same. That is, camera C
, The shooting target s can be changed in four different directions (D1, D2, D
Shoot from 3, D4). The imaging target s has four side surfaces a, b, c, d and an upper surface e, and FIGS. 16 to 19 show images taken from directions D1 to D4, respectively.

In this conventional image display method, for example, when it is desired to view an image from a direction D5 (indicated by a dashed line in FIG. 15), an image viewed from a direction closest to the direction D5, that is, from the direction D1 is viewed. The video (that is, the video of the shooting target s captured from the direction D1 by the camera C) is displayed on a monitor screen or the like. Therefore, the image viewed from the direction D5 is not reproduced.

[0005]

SUMMARY OF THE INVENTION An object of the present invention is to provide a method and apparatus for displaying a three-dimensional image by stereo photography, which can realistically reproduce a three-dimensional image of a subject from a captured two-dimensional image as an image viewed from an arbitrary viewpoint on a monitor screen. The purpose is to provide.

[0006]

SUMMARY OF THE INVENTION A method for displaying a three-dimensional image by stereo photography according to the present invention includes the steps of obtaining a pair of stereo images relating to a target object to be photographed; Obtaining the distance from the viewpoint origin in one of the stereo images of each point, and converting the obtained point distance data group into three-dimensional rectangular coordinates; and the point distance data group and boundary data for each area on the same plane. Generating a curved surface for interpolating each region for each region on the basis of the following: a corresponding pixel of one stereo image of the pair of stereo images on each of the plurality of generated curved surfaces; Generating a three-dimensional image of the object to be photographed by pasting in units.

Further, the three-dimensional image display apparatus for stereo photography according to the present invention comprises: means for obtaining a pair of stereo images relating to a target photographing object; based on data of the obtained pair of stereo images, Means for obtaining the distance from the viewpoint origin in one of the stereo images of each point, and converting the obtained point distance data group into three-dimensional rectangular coordinates; and the point distance data group and boundary data for each area on the same plane Means for generating a curved surface for interpolating each region for each region on the basis of the following: a corresponding pixel of one stereo image of the pair of stereo images on each of the plurality of generated curved surfaces in pixel units Means for generating a three-dimensional image of the object to be photographed by pasting the object.

[0008]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below based on illustrated embodiments. FIG. 1 shows a system configuration of one embodiment (first embodiment) of a three-dimensional image display device to which the present invention is applied. This device is a C that controls overall control of the device.
It has a PU 12, a digital camera interface 11 electrically connected to the CPU 12, and a stereo digital camera 10 connected to the interface 11. The apparatus has a RAM 13, a ROM 14, a keyboard 15, a mouse 16, and a CRT 17, which are electrically connected to the CPU 12, respectively.

The stereo digital camera 10 is composed of a pair of digital cameras (digital still cameras) 10a and 10b (see FIG. 3) whose photographing optical axes are separated from each other by a predetermined distance. Are included so that they can be photographed from different viewpoints. Each of the digital cameras 10a and 10b uses a CCD as an image sensor, and outputs the obtained image data (digital image signal) to the interface 11 in a serial manner in response to a request after imaging.

The image data output to the interface 11 is temporarily stored in a RAM 13 via a CPU 12, and subsequently, a predetermined control program, a keyboard 15 and a mouse 16
The CPU 12 performs predetermined image processing (stereoscopic image processing) on the image data stored in the RAM 13 in accordance with a command or the like input from the CRT 17, and then outputs the image data on which the image processing has been performed to the CRT 17 as a video signal.
Display a stereoscopic image on top.

FIG. 2 is a flowchart showing the processing operation of the CPU 12. Hereinafter, the processing operation of the CPU 12 will be described based on this flowchart.

First, image data (stereo image data) of a stereo photograph taken from the stereo digital camera 10 is obtained (step S10). FIG. 3 shows a state of photographing when the image data is input, and FIG.
2 shows a pair of image data obtained by the shooting. FIG.
As shown in the figure, the target photographing target usually includes a subject S, a background B, and a bottom surface G such as the ground, a floor surface, and a photographing table upper surface. FIG. 4 shows stereo image data obtained by the pair of digital cameras 10a and 10b. An image D1 shown on the left of FIG. 4 is an image (image data) of the photographing target shown in FIG. ) And FIG.
An image D2 shown on the right of FIG. 3 is an image (image data) of the shooting target shown in FIG. 3 obtained by the digital camera 10b.
In the image processing of this apparatus, the image D1 is treated as a main image, and the image D2 is treated as a sub image.

Next, a distortion correction (distortion correction) process is performed on each of the obtained image data D1 and D2 (step S12). Thereafter, based on the pair of image data D1 and D2, each of the photographed objects is subjected to a correlation method. The distance from the viewpoint origin of the point is obtained (step S14).

Subsequently, coordinate data (coordinate data on the CCD or pixel coordinate data) of each pixel on the boundary in the image D1 as the main image is obtained (step S1).
6). FIG. 5 shows the obtained coordinate data in three-dimensional orthogonal coordinates (XYZ coordinates). Thereafter, the distance data (coordinate data) of each point of the photographing target is edited and corrected (step S18). In this editing and correction work, interpolation distance data for interpolating the distance data of each point of the shooting target is created. FIG. 6 shows an image D1 including the interpolation distance data.
3 shows distance data.

Subsequently, an equation for generating a curved surface (interpolated curved surface) for each region is determined based on the point distance data group and the boundary data for each region on the same surface. (Step S20). Then, based on the obtained equation (a sequence of coefficient values at each order) and the boundary data,
Each pixel of the image D1, which is the main image, is pasted on the corresponding interpolated curved surface, thereby creating stereoscopic image data to be captured (step S22). FIG. 7 shows the state of the image data after generating the interpolated curved surface for all the regions. Bs in FIG. 7 indicates a portion of the background B that becomes a shadow of the subject S.

The created three-dimensional image data is output to the CRT 17 as a video signal and displayed on the CRT 17 as a three-dimensional image. Thereafter, the user inputs the observation direction (CRT 17) from the desired viewpoint through the keyboard 15 or the mouse 16.
(Display angle of the stereoscopic image to be displayed above), the shape of each interpolated curved surface when the object to be shot is viewed from the viewing direction from the desired viewpoint is calculated, and the viewing angle from the desired viewing direction is calculated based on the calculation result. The three-dimensional image in the case of being displayed is displayed on the CRT 17 (steps S24 and S26). Thereafter, when the user inputs the desired observation direction again via the keyboard 15 or the mouse 16, the processing of steps S24 and S26 is executed again, and the stereoscopic image obtained when the user views from the observation direction from the desired viewpoint is displayed on the CRT 17. Display.

FIG. 8 is a flowchart showing the processing (corrected curved surface generation processing) of the CPU 12 in steps S16 to S22 in FIG. First, after the processing in step S14 shown in FIG. 2, a point distance data group and boundary data in each area are read (step S3).
0). The point distance data is the actual XYZ coordinates at the photographing origin. The boundary data is the first,
The first, second, and third boundary data include second and third data, and are determined by the following acquisition process.

In one image (main image) of a stereo image in which the viewpoint origin is defined, a portion which is a single surface is surrounded by a closed curve, and pixel coordinates passing through the closed curve (coordinates in pixel units on a two-dimensional image) ) Is the first data. In the main image, when all parts are covered with the closed curve, a surface number (closed curve number) is assigned, and the attribute of the common closed curve portion is set as another surface number for the two surfaces sharing the closed curve. This is the second data. Further, as the shape attribute at the boundary of each surface, the data is described as third data including a description of a connection mode (smoothly connected, simply touched, not touched, etc.) with an adjacent surface. The boundary data includes the first to third data.

After reading the above point distance data group and boundary data, the point distance data group is smoothed (data redistribution) by predetermined statistical processing, and an initial curved surface (a surface on a sphere or a point A triangle plane having three adjacent points in the distance data as vertices is generated (steps S32 and S34). Thereafter, an initial target point is set based on the actual coordinates on the initial surface corresponding to the point distance data group and the boundary data, and a surface (interpolated surface) that fits the initial target point is determined by the least square method based on the initial surface. (Steps S36 and S38).

Subsequently, an evaluation value (a value indicating the degree of fit) of the generated surface (interpolated surface) with respect to the initial target point is obtained, and from the evaluation value, the initial target point is again fitted based on the generated surface. It is determined whether or not a simple curved surface (interpolated surface) needs to be generated by the least squares method (step S4).
0). If it is determined that there is no need to generate a curved surface again, the processing of steps S42 and S44 is skipped, and the process proceeds to step S46. If it is determined that it is necessary, the process proceeds to the following step S42.

In step S42, a new target point based on the actual coordinates on the curved surface generated in step S40 is set. Subsequently, a curved surface (interpolated curved surface) that fits the newly set target point is generated by the least square method (step S44). Thereafter, the process of step S40 is executed again, and when it is determined that the curved surface needs to be generated again, the processes of steps S42 and S44 are repeated. By the processes in steps S38 to S44, the final equation of the interpolated surface corresponding to each area is determined for each area.

Hereinafter, the process of determining the equation of the interpolation surface will be described in detail. Let r be the position of the object point from the viewpoint origin determined in one of the stereo images. Also,
The pixel coordinates (CC) of the object point in the one image
(2D coordinates on D) is (x, y), and this (x, y)
Let f be an expression for obtaining the position r from the above. Then, the following equation (1) holds. r = f (x, y) (1) Expression f is an n-character equation related to (x, y) and represents a surface in a three-dimensional space.

An object point group (r ₁ , r ₂ ,...) On one surface
.., R _k ) can be expressed by the above equation (1), the above equation (1) can be said to be an equation representing the surface. For the above object point group, the form (coefficient) of f so that the following equation holds
Should be determined. f (x, y) = a 0 x n + a 1 x n-1 y + ···· + a n-1 xy n-1 + a n y n r 1 = f (x 1, y 1) = a 0 x 1 ^{_{n + ····· + a n y 1}} n r 2 = f (x 2, y 2) = a 0 x 2 n + ····· + a n y 2 n · · · r k = f (x k , Y _k ) = a ₀ x _k ⁿ +... + A _n y _k ⁿ

That is,

(Equation 1)

If R = PX F = PX-R, F is set to 0 (zero). Then, by the least square method, Φ (evaluation value) = FT · F
It is sufficient to find X which is the minimum value of (T is a transposed matrix). That is, the partial derivative ∂Φ / ∂X of Φ with respect to X is 0
X that satisfies (zero) may be obtained (∂Φ / ∂X = 0). Therefore, the vector X is a coefficient of the equation of the correction surface to be obtained. If the evaluation value Φ does not become sufficiently small, the calculation is repeated until X can be regarded as a solution by repeating the least squares method.

In step S46, the coefficients and boundary data of the equation of the interpolated curved surface corresponding to each area are transferred to the following three-dimensional image display program (steps S24 and S26).

By the above processing, the target photographing target photographed by the stereo digital camera 10 is realistically reproduced on the CRT 17 at the same resolution as the two-dimensional images photographed by the digital cameras 10a and 10b. Further, since the corresponding curved surface of the photographing target is expressed by an equation representing a curved surface (interpolated curved surface), the amount of video data can be reduced, and therefore, it is suitable as communication data.

FIGS. 9 and 10 show another embodiment (second embodiment) of the three-dimensional image display device to which the present invention is applied. In the first embodiment, a stereo digital camera 1
However, in the second embodiment, a stereo camera 100 including a pair of still cameras 100a and 100b is used.
Use Each of the still cameras 100a and 100b is a single-lens reflex camera or a lens shutter camera using a silver halide film. In the present embodiment, first, a target photographing target is photographed by the stereo camera 100, and then a pair of stereo photographs (a pair of prints) obtained by each of the still cameras 100a and 100b are read by the digital image scanner 110, respectively. Digital image (digital image signal) via interface 111 to C
Transfer to PU12. Other configurations are the same as those of the three-dimensional image display device of the first embodiment. According to this configuration, the same effect as that of the three-dimensional image display device of the first embodiment can be obtained.

In the second embodiment, each still camera 1
A configuration may be adopted in which a positive film or a negative film photographed at 00a and 100b is directly read by the digital image scanner 110.

FIG. 11 and FIG. 12 show three examples to which the present invention is applied.
13 shows another embodiment (third embodiment) of the three-dimensional image display device. In the first embodiment, a stereo digital camera 1
In the third embodiment, one video recorder (digital video recorder or analog video recorder) 200 is used. Video recorder 200
Is mounted on a pair of guide rails 210 extending substantially in the left-right direction with respect to the imaging target and parallel to each other, and is movable along the guide rails 210 between the positions a and b in FIG. .

In the present embodiment, first, the video recorder 2
00 is moved at a constant speed along the guide rail 210 to perform a shooting operation, and at the same time, the obtained video signal is temporarily stored in the digital image memory 209 at a constant period in synchronization with a clock by the equal interval timer 211, and subsequently, The stored video signal is transmitted to the interface 212.
Are sequentially transferred to the CPU 12. Other configurations are the first
This is the same as the three-dimensional image display device of the embodiment. According to this configuration, the same effect as the three-dimensional image display device of the first embodiment can be obtained, and further, a continuous image of a stereoscopic image from different observation directions can be displayed on the CRT 17.

FIG. 13 and FIG.
14 shows another embodiment (fourth embodiment) of the two-dimensional image display device. In the first embodiment, a stereo digital camera 1
Although 0 is used, a camera and an image scanner are not used in the fourth embodiment. In the fourth embodiment, using a vernier caliper, a micrometer, and the like, each of the imaging targets S, G, and S from the viewpoint corresponding to the main image and the sub image in the first embodiment.
A plurality of characteristic points and points on the outline of B are measured, and the measurement results are input to the CPU 12 via the keyboard 15. CPU
In step 12, the shape of each object to be photographed is obtained by obtaining the dimensions a to i (see FIG. 13) of each object to be photographed from the input measurement data. Other configurations are the same as those of the three-dimensional image display device of the first embodiment. According to this configuration, the same effect as the three-dimensional image display device of the first embodiment can be obtained.

[0033]

As described above, according to the present invention, a curved surface for interpolating each area based on the point distance data group and the boundary data for each area on the same plane is generated for each area. Since a corresponding pixel of one stereo image of the pair of stereo images is pasted on a pixel-by-pixel basis on each of the generated plurality of curved surfaces to generate a three-dimensional image of a shooting target, a three-dimensional image of a subject is generated. Can be realistically reproduced on a monitor screen at the same resolution as a captured two-dimensional image, and does not require a huge amount of video data.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a system configuration of a first embodiment of a three-dimensional image display device to which the present invention has been applied.

FIG. 2 is a flowchart showing an operation of a CPU of the three-dimensional image display device shown in FIG.

FIG. 3 is a diagram illustrating a state of shooting when inputting image data in the three-dimensional image display device according to the first embodiment.

FIG. 4 is a diagram showing a pair of image data obtained by photographing with the pair of digital cameras shown in FIG. 3;

5 is a diagram showing coordinate data of each pixel on a boundary in the main image shown in FIG. 4 in three-dimensional orthogonal coordinates.

6 is a diagram showing coordinate data obtained by adding interpolation distance data to the coordinate data shown in FIG. 5 in three-dimensional orthogonal coordinates.

FIG. 7 is a diagram illustrating, using three-dimensional orthogonal coordinates, a state of image data after an interpolation surface is generated for all regions.

FIG. 8 is a flowchart illustrating a correction surface generation process.

FIG. 9 is a diagram illustrating a state of shooting at the time of inputting image data in a second embodiment of a three-dimensional image display device to which the present invention has been applied.

FIG. 10 is a block diagram illustrating a system configuration according to a second embodiment.

FIG. 11 shows a third example of the three-dimensional image display device to which the present invention is applied.
FIG. 5 is a diagram illustrating a state of shooting when inputting image data in the embodiment.

FIG. 12 is a block diagram illustrating a system configuration according to a third embodiment.

FIG. 13 shows a fourth example of the three-dimensional image display device to which the present invention is applied.
FIG. 4 is a diagram illustrating a state when image data is input in the embodiment.

FIG. 14 is a block diagram illustrating a system configuration according to a fourth embodiment.

FIG. 15 is a diagram showing a shooting situation by a conventional image display method.

FIG. 16 is a diagram showing a video when a shooting target is shot from a direction D1 shown in FIG. 15;

FIG. 17 is a diagram showing a video when a shooting target is shot from a direction D2 shown in FIG. 15;

FIG. 18 is a diagram showing a video when a shooting target is shot from a direction D3 shown in FIG.

FIG. 19 is a diagram showing a video when a shooting target is shot from a direction D4 shown in FIG. 15;

[Explanation of symbols]

 10 Stereo digital camera 10a 10b Digital camera S Subject B Background G Bottom

────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] November 18, 1998

[Procedure amendment 1]

[Document name to be amended] Statement

[Correction target item name] Claim 1

[Correction method] Change

[Correction contents]

[Procedure amendment 2]

[Document name to be amended] Statement

[Correction target item name] 0013

[Correction method] Change

[Correction contents]

Next, a distortion correction (distortion correction) process is performed on each of the obtained image data D1 and D2 (step S12). Thereafter, based on the pair of image data D1 and D2, the subject to be photographed is subjected to triangulation. The distance of each point from the viewpoint origin is obtained (step S14).

[Procedure amendment 3]

[Document name to be amended] Drawing

[Correction target item name] Figure 2

[Correction method] Change

[Correction contents]

FIG. 2

[Procedure amendment 4]

[Document name to be amended] Drawing

[Correction target item name] Fig. 4

[Correction method] Change

[Correction contents]

FIG. 4

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁶ Identification code FI H04N 13/04 G06F 15/62 350V

Claims (5)

[Claims] A step of obtaining a pair of stereo images relating to a target photographing target; and a distance from a viewpoint origin in one of the stereo images of each point of the photographing target based on data of the obtained pair of stereo images. And converting the obtained point distance data group into three-dimensional orthogonal coordinates; and forming a curved surface that interpolates each area based on the point distance data group and boundary data for each area on the same plane. Generating a three-dimensional image of the object to be photographed by pasting the corresponding pixel of one of the stereo images of the pair of stereo images on a pixel-by-pixel basis on each of the generated plurality of curved surfaces; And displaying a three-dimensional image by stereo photography. 2. The method according to claim 1, further comprising a step of calculating a shape of each of the plurality of curved surfaces when the image is viewed from an arbitrary viewing direction. 3D image display method. 3. The method according to claim 1, wherein the step of generating a plurality of curved surfaces includes the step of interpolating each region based on a group of point distance data and boundary data for each region. And a method for displaying a three-dimensional image by stereo photography. 4. The method according to claim 1, wherein the boundary data is obtained in one image of a stereo image defining a viewpoint origin.
Surrounding a portion to be one surface with a closed curve, and assigning a surface number when the first data relating to the pixel coordinates passing through the closed curve and all the portions in the one stereo image are covered by the closed curve. The second data that has the attribute of the common closed curve portion of the two surfaces sharing the other surface number as another surface number, and the third attribute including the description of the connection form with the adjacent surface as the shape attribute at the boundary of each surface A method for displaying a three-dimensional image by stereo photography, comprising: 5. A means for obtaining a pair of stereo images relating to a target photographing target; and a distance from the viewpoint origin in one of the stereo images of each point of the photographing target based on the data of the obtained pair of stereo images. Means for converting the obtained point distance data group into three-dimensional rectangular coordinates; and forming a curved surface for interpolating each area based on the point distance data group and boundary data for each area on the same plane. Means for generating for each region; 3 pixels of the object to be photographed by pasting corresponding pixels of one stereo image of the pair of stereo images on each of the plurality of generated curved surfaces in units of pixels.
Means for generating a three-dimensional image.