JPH11355806A - Method and device for three-dimensional image display by stereoscopic image photographing - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realistically reproduce the three-dimensional picture of an object on a monitoring screen as an image viewed from an arbitrary viewpoint from two-dimensional images which are image-picked up by providing a step or the like for obtaining the plural pairs of stereo pictures on an objective photographing object from a plurality of different viewpoints. SOLUTION: A solid image generated based on a pair of image data obtained in photographing from any one viewpoint among a plurality of viewpoints or is outputted to a CRT 17 as a video signal and it is displayed on the CRT 17 as a stereoscopic image. When a user designates an observation direction from the desired viewpoint through a keyboard 15 or a mouse 16, the forms of respective faces in a case when the viewpoint nearest this observation direction is selected among a plurality of viewpoints, and a photographing object is viewed from the observation direction are calculated. The stereoscopic image viewed from the desired observation direction based on the calculation results is displayed on the CRT 17. Thus, the photographing object which is image-picked up by a stereo digital camera 10 is reproduced realistically with resolution in a degree similar to a two-dimensional image which is image- picked up.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for displaying a three-dimensional image by stereo photography and an apparatus using the method.

[0002]

2. Description of the Related Art Conventionally, a subject to be photographed (subject)
As a display method for displaying the image on a monitor screen such as a CRT, a photographing target is first photographed (imaged) from each of a plurality of different positions around the image, and then an image closest to a desired observation direction (angle) is displayed on the monitor screen. An image display method to be displayed on the top is known.

In such a conventional image display method, since a desired observation direction depends on the direction at the time of photographing, an image viewed from an arbitrary viewpoint cannot be reproduced. FIG. 17 shows an example of a shooting situation by the conventional image display method. In this example, the same camera (digital camera) C captures a substantially cuboid-shaped imaging target s from each of a plurality of different positions (four locations) around the same. That is, camera C
, The photographing target s can be moved in four different directions (D1, D2, D
Shoot from 3, D4). The photographing target s has four side surfaces a, b, c, d and a top surface e, and FIGS. 18 to 21 show images when photographed from directions D1 to D4, respectively.

In this conventional image display method, for example, when it is desired to view an image from a direction D5 (indicated by a dashed line in FIG. 17), an image viewed from a direction closest to the direction D5, that is, from the direction D1. The video (that is, the video of the shooting target s captured from the direction D1 by the camera C) is displayed on a monitor screen or the like. Therefore, the image viewed from the direction D5 is not reproduced.

[0005]

SUMMARY OF THE INVENTION An object of the present invention is to provide a method and apparatus for displaying a three-dimensional image by stereo photography, which can realistically reproduce a three-dimensional image of a subject from a captured two-dimensional image as an image viewed from an arbitrary viewpoint on a monitor screen. The purpose is to provide.

[0006]

SUMMARY OF THE INVENTION A method of displaying a three-dimensional image by stereo photography according to the present invention includes the steps of obtaining a plurality of pairs of a pair of stereo images of a target object from a plurality of different viewpoints; Obtaining a point distance data group by obtaining a distance from a viewpoint origin in one of the stereo images of each point of the photographing object based on the data of the stereo image of each pair, for each pair of stereo images; For each pair of stereo images, converting the point distance data group relating to the one stereo image defining the viewpoint origin to a single reference coordinate having a predetermined point as the origin; Determining the boundaries of the regions constituting one surface for each region in the stereo image of the above; and a plurality of the above obtained between different viewpoints A step of comparing regions determined by the boundary between quasi-coordinates to determine whether they are the same region; each region determined to be the same by this determination and the point distance included in the boundary Generating a curved surface for interpolating each region on the basis of the data group for each region; and providing, on each of the plurality of generated curved surfaces, a viewpoint in which the projected area of the pair of stereo images is the largest. Generating a three-dimensional image of the object to be photographed by pasting the corresponding pixels of the one stereo image of the pair of stereo images from each other on a pixel-by-pixel basis; A step of selecting a close viewpoint; with respect to the three-dimensional image composed of the plurality of curved surfaces obtained from the selected viewpoint, when the three-dimensional image is viewed from any observation viewpoint. It is characterized by having: a step of calculating a shape of each of the plurality of curved surfaces.

Further, the three-dimensional image display apparatus using stereo photography according to the present invention includes: means for obtaining a plurality of pairs of stereo images relating to a target photographing object from a plurality of different viewpoints;
For each pair of stereo images obtained from the plurality of pairs of stereo images, the distance from one viewpoint of the stereo image of each of the shooting target points to the point is determined based on the data of each pair of stereo images. Means for obtaining a distance data group; means for converting the point distance data group for the one stereo image defining the viewpoint origin for each pair of stereo images into a single reference coordinate having a predetermined point as the origin Means for determining, for each region, a boundary of a region constituting one surface in the one stereo image in which the viewpoint origin is determined; and a plurality of reference coordinates obtained between different viewpoints, Means for comparing the determined areas with each other to determine whether they are the same area; each area determined to be the same by this determination and included in its boundary Based on the serial point distance data group,
Means for generating, for each region, a curved surface for interpolating each region; and a pair of stereo images from the viewpoint where the projected area is the largest of the pair of stereo images on each of the plurality of generated curved surfaces. Means for generating a three-dimensional image of the object to be photographed by pasting corresponding pixels of the one stereo image in pixel units; means for selecting a viewpoint closest to a desired observation direction from the plurality of different viewpoints Means for calculating a shape of each of the plurality of curved surfaces when the three-dimensional image is viewed from an arbitrary observation viewpoint with respect to the three-dimensional image formed of the plurality of curved surfaces obtained from the selected viewpoint;
It is characterized by having.

[0008]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below based on illustrated embodiments. FIG. 1 shows a system configuration of an embodiment of a three-dimensional image display device to which the present invention is applied. This device has a CPU 12 that controls the entire control of the device, a digital camera interface 11 electrically connected to the CPU 12, and a stereo digital camera 10 connected to the interface 11. In addition, the present apparatus has RAs electrically connected to the CPU 12, respectively.
M13, ROM 14, keyboard 15, mouse 1
6 and a CRT 17.

The stereo digital camera 10 is composed of a pair of digital cameras (digital still cameras) 10a and 10b (see FIG. 4) whose photographing optical axes are separated from each other by a fixed distance. Are included in a plurality of different viewpoints (viewpoint 1, viewpoint 1).
An image is taken from each of the viewpoints 2,..., Viewpoint n). Each of the digital cameras 10a and 10b uses a CCD as an image sensor, and outputs the obtained image data (digital image signal) to the interface 11 in a serial manner in response to a request after imaging.

The image data output to the interface 11 is temporarily stored in a RAM 13 via a CPU 12, and subsequently, a predetermined control program, a keyboard 15 and a mouse 16
The CPU 12 performs predetermined image processing (stereoscopic image processing) on the image data stored in the RAM 13 in accordance with a command or the like input from the CRT 17, and then outputs the image data on which the image processing has been performed to the CRT 17 as a video signal.
Display a stereoscopic image on top.

FIGS. 2 and 3 are flowcharts showing the processing operation of the CPU 12. Hereinafter, the processing operation of the CPU 12 will be described based on this flowchart.

First, at each of a plurality of different viewpoints (viewpoint 1, viewpoint 2,..., Viewpoint n), image data (stereo image data) of each stereo photograph taken by the stereo digital camera 10 is obtained (step S1).
0). FIG. 4 shows a state of shooting at the time of inputting the image data. 5 and 6 show a pair of stereo image data obtained by shooting from the viewpoint 1 shown in FIG. 4, and FIGS. 7 and 8 show a pair of stereo image data obtained by shooting from the viewpoint 2 shown in FIG. 3 shows stereo image data. 9 and 10 show a pair of stereo image data obtained by shooting from the viewpoint n shown in FIG.

As shown in FIG. 4, a target photographing target usually includes a subject S, a background B, and a bottom surface G such as the ground, a floor surface, and a photographing table upper surface. An image D1 shown in each of FIGS. 5, 7, and 9 is an image (image data) of the photographing target shown in FIG. 4 obtained by the digital camera 10a.
An image D2 shown in each of FIGS. 8 and 10 is an image (image data) of the shooting target shown in FIG. 4 obtained by the digital camera 10b. The image processing of the present apparatus handles the image D1 as a main image and the image D2 as a sub-image with respect to a stereo image obtained at each viewpoint.

The processing from step S12 to step S18 following step S10 is performed by a pair of stereo image data (image data D1, D1) obtained at each of a plurality of different viewpoints (viewpoint 1, viewpoint 2,..., Viewpoint n). D2) are performed in the order of viewpoint 1, viewpoint 2,..., Viewpoint n. Therefore, after the processing in step S10, first, a pair of stereo image data (image data D
(1, D2), the processes from steps S12 to S18 are performed.

In step S12, a distortion correction (distortion correction) process is performed on the obtained pair of stereo image data (image data D1 and D2). After that, based on the pair of image data D1 and D2, the distance from the viewpoint origin in the main image D1 of each point to be photographed is obtained by the correlation method (step S14).

Subsequently, in the image D1 which is the main image, the boundaries of the regions constituting one surface are determined for each region,
The coordinate data (CCD coordinate data or pixel coordinate data) of each pixel on each boundary is obtained, and the obtained coordinate data is converted into three-dimensional rectangular coordinates (XY
(Z coordinate) (step S16).

After that, the distance data (point distance data, that is, coordinate data) of each point of the photographing object is edited and corrected (step S18). In this editing correction processing, an interpolated distance data group for interpolating the point distance data group included in each area and its boundary is created. (1)-of FIG.
(7) is a distance data group (viewpoint 1) of each area (areas a, b, c, d, e, f, g) including this interpolation distance data group.
Are shown in three-dimensional orthogonal coordinates.

Next, each viewpoint (viewpoint 1, viewpoint 2,...,
It is determined whether the processing from step S12 to step S18 has been performed for all of the pair of image data D1 and D2 at the viewpoint n) (step S20). If not, the process returns to step S12, and the steps S12 to S1 are performed for a pair of image data D1 and D2 at the next viewpoint (for example, viewpoint 2 after viewpoint 1).
8 is performed, and if all the processes have been performed, the process proceeds to the subsequent step S22.

That is, from step S12 to step S18
Is performed similarly for a pair of stereo image data obtained at each viewpoint in the order of viewpoint 1, viewpoint 2,..., Viewpoint n. Therefore, in the processing of step S16, viewpoints 1 to
Regarding the coordinate data of the main image D1 of each viewpoint of the viewpoint n,
The coordinate data of each pixel on each boundary is converted into a single three-dimensional orthogonal coordinate.

After obtaining the distance data group of each area including the interpolation distance data group for the viewpoint 1 by the processing of steps S12 to S18, the distance of each area including the interpolation distance data group for the viewpoint 2 is obtained by the same processing. Get the data group. (1) to (7) of FIG. 12 show each area (areas h, b, c, d, e, f,
The distance data group (i) (the distance data group obtained from the viewpoint 2) is indicated by three-dimensional orthogonal coordinates. Thereafter, a distance data group is similarly obtained for each viewpoint up to viewpoint n.

It should be noted that the region a shown in FIG.
The region h shown in (1) is a region corresponding to the left and right sides of the background (background object) B shown in each of FIGS. Therefore, the region h is not included in the plurality of regions of the image obtained from the viewpoint 1 shown in FIG.
Since the region a on the side opposite to the region h is not shown in the plurality of regions of the image obtained from, the region a is not included. Similarly, a region g shown in FIG. 11 (7) and a region i shown in FIG. 12 (7) are regions corresponding to the left and right sides of the bottom surface G of the imaging table or the like shown in each of FIGS. is there. Therefore, since the region i on the opposite side to the region g is not shown in the plurality of regions of the image obtained from the viewpoint 1 shown in FIG. 11, the region i is not included, and from the viewpoint 2 shown in FIG. The region g is not included in the plurality of regions of the obtained image because the region g opposite to the region i is not shown.

In the following step S22, the same region is determined by comparing the respective regions between the viewpoints (viewpoint 1, viewpoint 2,..., Viewpoint n). For example, between the image data of viewpoint 1 and viewpoint 2, each region of viewpoint 1 shown in FIG. 11 is compared with each region of viewpoint 2 shown in FIG.
As a result, the areas a and g of the viewpoint 1 and the areas h and i of the viewpoint 2
Are determined not to have the same region, and the viewpoint 1
Is determined to be the same as the region b of the viewpoint 2, the region c of the viewpoint 1 is determined to be the same as the region c of the viewpoint 2, and similarly, each of the regions d, e, and f of the viewpoint 1 is The area is determined to be the same as the corresponding area d, e, f of the viewpoint 2.

A point distance data group for each of the areas (b, c, d, e, f) determined to be the same area and the areas (a, g, h, i) not having the same area is determined. Based on and the boundary data, an equation for generating a curved surface (interpolated curved surface) for each region for each of all viewpoints is determined (step S24). Then, each pixel of the image D1, which is the main image, is pasted on the corresponding interpolated curved surface based on the obtained equation (a sequence of coefficient values in each order) and the boundary data. Stereoscopic image data (3
(Step S26). At this time,
The corresponding pixel of the main image D1 at the viewpoint having the largest projection area is pasted on a pixel-by-pixel basis on the generated interpolation curved surface, thereby generating stereoscopic image data (three-dimensional image) to be captured. (1) to (9) in FIG. 13 show distance data groups (viewpoint 1 and viewpoint 2) of each area a to i including the interpolation distance data group.
14 are respectively shown by a single three-dimensional orthogonal coordinate, and (1) to (9) in FIG.
The image data after generating the interpolation surface with respect to
They are shown in dimensional orthogonal coordinates.

Thereafter, a plurality of viewpoints (viewpoint 1, viewpoint 2,...)
.. any one of viewpoints (for example, viewpoint 1)
3D image data created based on a pair of image data D1 and D2 obtained by shooting from
And displayed as a stereoscopic image on the CRT 17 (step S28). Then, the user enters the keyboard 15
And the observation direction (CR
When a display angle of a stereoscopic image to be displayed on T17 is specified, a viewpoint closest to the specified observation direction (observation viewpoint) is selected from a plurality of viewpoints (Steps S30 and S3).
2). Then, the shape of each surface when the photographing target is viewed from the observation direction from the selected viewpoint is calculated, and a stereoscopic image when viewed from the desired observation direction is displayed on the CRT 17 based on the calculation result (step S34). ). Thereafter, when the user again inputs the desired observation direction via the keyboard 15 or the mouse 16, steps S32 and S3 are performed.
The processing of step 4 is executed again, and the stereoscopic image viewed from the observation direction from the desired viewpoint is displayed on the CRT 17. FIG.
Is a CRT 17 when the photographing target is viewed from a certain observation direction.
FIG. 16 shows the stereoscopic image displayed above, and FIG.
2 shows a stereoscopic image displayed on the CRT 17 when the photographing target is viewed from an observation direction different from the observation direction of FIG.

By the above processing, the target photographing target photographed by the stereo digital camera 10 is realistically reproduced on the CRT 17 with the same resolution as the two-dimensional images photographed by the digital cameras 10a and 10b. Further, image data from a viewpoint closest to the desired observation direction is selected, and an image viewed from the desired observation direction is displayed based on the selected image data. The shadow area is reduced. In addition, when stereoscopic photography is performed, if the photographing target is uniformly photographed from many different viewpoints, a shadow portion cannot be formed on the photographing target displayed on the CRT 17 even when the observation direction is changed.

[0026]

As described above, according to the present invention, a three-dimensional image of a subject can be realistically reproduced on a monitor screen with the same resolution as a captured two-dimensional image. If the photographing target is photographed evenly from many different viewpoints, a shadow portion cannot be formed on the photographing target displayed on the CRT even if the observation direction is changed.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a system configuration of an embodiment of a three-dimensional image display device to which the present invention has been applied.

FIG. 2 is a flowchart showing an operation of a CPU of the three-dimensional image display device shown in FIG.

FIG. 3 is a flowchart illustrating an operation of a CPU of the three-dimensional image display device illustrated in FIG. 1;

FIG. 4 is a diagram showing a state of shooting when inputting image data in the three-dimensional image display device.

FIG. 5 is a diagram showing image data obtained by shooting from a viewpoint 1 by one of a pair of digital cameras.

FIG. 6 is a diagram showing image data obtained by shooting from the viewpoint 1 by the other of the pair of digital cameras.

FIG. 7 is a diagram showing image data obtained by shooting from a viewpoint 2 by one of a pair of digital cameras.

FIG. 8 is a diagram illustrating image data obtained by shooting from the viewpoint 2 by the other of the pair of digital cameras.

FIG. 9 is a diagram showing image data obtained by shooting from a viewpoint n by one of a pair of digital cameras.

FIG. 10 is a diagram showing image data obtained by shooting from a viewpoint n by the other of a pair of digital cameras.

FIG. 11 is a diagram showing, using three-dimensional rectangular coordinates, a distance data group of each region including an interpolation distance data group obtained from image data from a viewpoint 1;

FIG. 12 is a diagram illustrating, using three-dimensional orthogonal coordinates, a distance data group of each region including an interpolation distance data group obtained from image data from a viewpoint 2.

FIG. 13 is obtained from image data from viewpoint 1 and viewpoint 2,
FIG. 6 is a diagram illustrating a distance data group of each region including an interpolation distance data group in a single three-dimensional orthogonal coordinate system.

FIG. 14 is a diagram illustrating image data after generating an interpolated curved surface for each region in a single three-dimensional orthogonal coordinate system.

FIG. 15 is a CR when a photographing target is viewed from a certain observation direction.
FIG. 9 is a diagram illustrating a stereoscopic image displayed on T.

FIG. 16 is a diagram showing a stereoscopic image displayed on a CRT when the photographing target is viewed from an observation direction different from the observation direction in FIG.

FIG. 17 is a diagram showing a shooting situation by a conventional image display method.

18 is a diagram illustrating an image when an object to be photographed is photographed from a direction D1 shown in FIG. 17;

FIG. 19 is a diagram showing a video when a shooting target is shot from a direction D2 shown in FIG. 17;

20 is a diagram showing an image when an object to be photographed is photographed from a direction D3 shown in FIG. 17;

21 is a diagram illustrating an image when an object to be photographed is photographed from a direction D4 shown in FIG. 17;

[Explanation of symbols]

 10 Stereo digital camera 10a 10b Digital camera S Subject B Background G Bottom

Claims (4)

[Claims] 1. obtaining a plurality of pairs of stereo images of a target photographing object from a plurality of different viewpoints;
For each pair of stereo images obtained from the plurality of pairs of stereo images, the distance from one viewpoint of the stereo image of each of the shooting target points to the point is determined based on the data of each pair of stereo images. Obtaining a distance data group; and converting the point distance data group relating to the one stereo image defining the viewpoint origin for each pair of stereo images into a single reference coordinate having a predetermined point as the origin. And; in the one stereo image defining the viewpoint origin, defining a boundary of an area constituting one surface for each area; and, between a plurality of reference coordinates obtained between different viewpoints, Comparing the determined areas with each other to determine whether they are the same area; and determining each area determined to be the same by this determination. Generating, for each region, a curved surface that interpolates each region based on the point distance data group included in the boundary; and on each of the plurality of generated curved surfaces, A step of pasting corresponding pixels of the one stereo image of the pair of stereo images from the viewpoint at which the projection area is maximized in pixel units to generate a three-dimensional image of the imaging target; from the plurality of different viewpoints, Selecting a viewpoint closest to a desired observation direction; and, for a three-dimensional image composed of the plurality of curved surfaces obtained from the selected viewpoint, the plurality of viewpoints when the three-dimensional image is viewed from any observation viewpoint. Calculating a shape of each of the curved surfaces; a method for displaying a three-dimensional image by stereo photography. 2. The method according to claim 1, further comprising the step of determining whether or not there is the same area by the step of determining whether or not there is the same area. A three-dimensional image display method using stereo photography, comprising a step of generating a curved surface for interpolating each region for each region based on the included point distance data group. 3. A means for obtaining a plurality of pairs of stereo images related to a target photographing object from a plurality of different viewpoints; and for each pair of stereo images obtained from the obtained plurality of pairs of stereo images, each pair of stereo images. Means for obtaining a point distance data group by obtaining a distance from the viewpoint origin in one of the stereo images of each point of the photographing object based on the data of the above; Means for converting the point distance data group relating to the stereo image into a single reference coordinate having a predetermined point as the origin; and a boundary of an area constituting one surface in the one stereo image defining the viewpoint origin. Means for determining each of the regions; and between the plurality of reference coordinates obtained between different viewpoints, the regions determined by the boundary are compared with each other to be the same region Means for determining whether or not each of the areas is determined to be the same; and a curved surface for interpolating each area based on each of the areas determined to be the same and the above-mentioned point distance data group included in the boundary. Means for generating, on each of the generated plurality of curved surfaces, a corresponding pixel of the one stereo image of the pair of stereo images from the viewpoint where the projected area is the largest among the pair of stereo images. Means for generating a three-dimensional image of the photographing target by pasting in units; means for selecting a viewpoint closest to a desired observation direction from the plurality of different viewpoints; and a plurality of viewpoints obtained from the selected viewpoint. Means for calculating the shape of each of the plurality of curved surfaces when the three-dimensional image is viewed from an arbitrary observation viewpoint for a three-dimensional image formed of a curved surface. 3-dimensional image display apparatus according to a. 4. The apparatus according to claim 3, further comprising, for each of the areas determined to have no identical area by the means for determining whether the area is the same area, the area and its boundary. A three-dimensional image display apparatus using stereo photography, comprising: means for generating a curved surface for interpolating each area for each area based on the included point distance data group.