JPH11220758A - Method and device for stereoscopic image display - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a stereoscopic vision of a wide visual field image, such as a 360 degrees panoramic image. SOLUTION: For example, omniazimuth image data from four image-pickup points P1, P2, P3 and P4 that are slightly deviated from one another are prepared. When the line of sight is within a range of 3±22.5 degrees, parts near a D3 direction of the omniazimuth image data from the image-pickup point P1 are shown to the left eye of an observer, while parts near the D3 direction of the omniazimuth image data from the image-pickup point P2 are shown to the right eye of the observer. By switching the omniazimuth image data shown to the left and right eyes, in accordance with the change in the line of sight, the omniazimuth can be viewed stereoscopically without disconnection. The omniazimuth picture data for example, are prepared by projecting on a ball surface the image picked up from the same point by a digital camera or the like, pasting then together to composite them, and are retained as stenographic projecting data.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for realizing stereoscopic viewing of a wide-field image.

[0002]

2. Description of the Related Art As a method of realizing a stereo (stereoscopic) image display based on a two-dimensional image captured by a digital camera or the like, a set of two images having different viewpoints is prepared.
In general, a method of showing images of different viewpoints to the left and right eyes, respectively. However, simply preparing two images covers a wide viewing angle, for example, 360
It was impossible to realize stereoscopic viewing of a panoramic image. In order to cover a wide viewing angle, two
It is necessary to prepare a set of images and switch the set of images to be shown to the observer according to the azimuth. For example, for stereoscopic viewing of a 360-degree panoramic image, it was necessary to prepare 15 sets (30 images) or more.

As a related technique, a left-eye image and a right-eye image are created by extracting a front image from a two-dimensional image, moving the left-right image relative to a background image, and enlarging it, and displaying them in a three-dimensional image. A method for realizing a stereo image display by outputting to a device is disclosed in
No. 2533.

[0004]

As described above, in the method of preparing two sets of images for each azimuth and switching the set of images to be shown to the observer according to the azimuth, the azimuth changes. There is a problem that the visual field is sometimes interrupted. That is, the surroundings cannot be continuously viewed in three dimensions.

It is an object of the present invention to provide means for enabling stereoscopic viewing of a wide-field image such as a 360-degree panoramic image without causing such a break in the visual field.

[0006]

In order to achieve the above-mentioned object, in the present invention, three or more omnidirectional image data whose shooting points are shifted are prepared. Each omnidirectional image data is, for example, a large number of image data taken while changing the shooting direction along a horizontal plane with a digital camera etc. from a fixed point, projected on a spherical or cylindrical surface centering on the shooting point and pasted The image data includes 360-degree panoramic image data combined and combined. Then, in the present invention, the omnidirectional image data for the left eye and the omnidirectional image data for the right eye are selected from the three or more omnidirectional image data based on the direction of the line of sight of the observer. By showing the image as an image to the left and right eyes of the observer, stereoscopic viewing of a wide-field image such as a 360-degree panoramic image is realized.

[0007]

Embodiments of the present invention will be described below with reference to the accompanying drawings.

In the present invention, at least three pieces of omnidirectional image data in which the photographing points are shifted are prepared. An example of a method of creating the omnidirectional image data will be described with reference to FIG.

When creating omnidirectional image data, a large number of images are photographed from a fixed photographing point using a digital camera or the like while changing photographing directions so as to make a round along a horizontal plane, for example. And according to one embodiment of the present invention,
For example, by projecting each image data onto a spherical surface A having a center O as the center of the lens of a digital camera or the like and a distance between the lens and the imaging surface, or a radius of an arbitrary magnification, and bonding and combining them, Omnidirectional image data is created. As described in detail in the specification of Japanese Patent Application Nos. 9-303893 and 9-234095 of the applicant of the present invention, such a composite image is obtained by combining pixels in the overlap area of adjacent image data. The conversion matrix of the azimuth angle coordinate between the images is obtained by using the relative positional relationship described above, and each image data is converted into the unified azimuth angle coordinate by using the matrix. If you want to realize a stereoscopic view of a three-dimensional panoramic image that covers the whole sky above the horizontal plane, when shooting an image with a digital camera or the like, you can also change the shooting direction in the vertical direction and shoot Then, the captured image data may be similarly pasted and combined on the spherical surface B.

In order to realize stereoscopic vision, it is necessary to store omnidirectional image data in a storage device or the like. According to one embodiment of the present invention, as described above, the omnidirectional image data projected onto the spherical surface and stitched and synthesized is applied to the present applicant's Japanese Patent Application No. Hei 9 (1994) -93207.
As detailed in the specification of US Pat. No. 2,340,955, it is stored as data projected onto a disk D on a horizontal plane C.

According to an embodiment of the present invention, for example, four omnidirectional image data in which the photographing points are shifted are created by the above-described spherical projection method, and each of them is stored in the form of plane projection data. Is done. This is shown in FIG. FIG.
, P1, P2, P3, and P4 respectively correspond to the photographing points of four omnidirectional image data. These shifts of the photographing point are determined in consideration of the parallax between the left and right eyes of a human, and are typically selected in a range of 10 cm to 50 cm. The omnidirectional image data from each of the photographing points P1, P2, P3, and P4 are respectively represented by circles Q1, Q2, and Q in FIG.
3, and are stored in the form of orthographic projection on the disk corresponding to Q4. These four omnidirectional image data are very similar to each other because the photographing points are close to each other.

According to one embodiment of the present invention, when the direction of the line of sight is specified, one omnidirectional data for the left eye and one omnidirectional data for the right eye are selected from the four stored omnidirectional image data. By selecting azimuth image data and displaying the corresponding omnidirectional image data at positions corresponding to the line of sight of the left and right eyes of the observer, stereoscopic viewing of a wide-field image is realized. This will be described with reference to FIG.

The sight line direction is, for example, ± 22.5 in the D3 direction.
If it is within the range of ゜, the omnidirectional image data from the shooting point P1 is selected as the omnidirectional image data for the left eye, and its D
The area near the three directions is shown to the left eye, and at the same time, the shooting point P
If the omnidirectional image data from 2 is selected as the omnidirectional image data for the right eye, and the region near the D3 direction is shown to the right eye, stereoscopic vision becomes possible. If the azimuth of the line of sight is within the range of ± 22.5 ° in the D4 direction, the omnidirectional image data from the photographing point P1 is selected as the omnidirectional image data for the left eye, and an area near the D4 direction is selected as the left eye. At the same time, the omnidirectional image data from the photographing point P4 is selected as the omnidirectional image data for the right eye, and an area near the D4 direction is shown to the right eye, whereby stereoscopic vision is possible. By changing the azimuth of the line of sight in this way, one omnidirectional image data observed by the observer is switched to another omnidirectional image data (in the case of the example described above, the omnidirectional image data is separated from the azimuth D3 + 22.5 °). The omnidirectional image data observed by the right eye is switched from that of the photographing point P2 to that of the photographing point P4), the omnidirectional image data observed by the left eye does not change, and the omnidirectional image data from the photographing point P2 is not changed. Since the data and the omnidirectional image data from the photographing point P4 are very similar, the observer hardly notices the switching of the omnidirectional image data.
Similarly, the observer can continuously and stereoscopically view a scene within a range of 360 degrees around the observer.

It is to be noted that the omnidirectional image data shown to the right eye or the left eye during the movement of the line of sight is not suddenly switched, but the two omnidirectional image data before and after the switching are overlapped and displayed continuously. May be switched.
For example, in FIG. 2, when the line-of-sight direction is sequentially changed from the D3 direction to the D4 direction, the photographing point P is set in an intermediate direction.
2, the omnidirectional image data and the omnidirectional image data from the photographing point P4 are overlapped and displayed, and are gradually switched to one omnidirectional image data. This makes it more difficult to notice the switching of the omnidirectional image data, thereby enabling a smoother omnidirectional image to be stereoscopically viewed.

When the omnidirectional image data projected in a plan view is shown to an observer, it is necessary to reproject the data onto the original spherical surface. When observing images on a head-mounted (goggles) type stereoscopic display device or a stereoscopic display device with a spherical display surface, the omnidirectional image data reprojected onto the spherical surface is displayed as it is. do it. However,
When observing an image on a display device having a flat display surface, it is better to reproject the omnidirectional image data onto a spherical surface and then project it on the display plane before displaying the image, since an image with less distortion can be observed. There will be.

According to another embodiment of the present invention, a large number of image data shot by a digital camera or the like from a fixed shooting point is converted into a central axis passing through a center O and perpendicular to a horizontal plane C as shown in FIG. And a cylindrical surface B having a radius equal to or equal to the distance between the lens center and the imaging surface, or having an arbitrary magnification.
, And the images are combined together to create omnidirectional image data. The omnidirectional image data is stored in a two-dimensional storage space corresponding to the developed view of the cylindrical surface B. Note that a pixel having a large azimuth from the horizontal plane C is projected at a position far away from the horizontal plane C on the cylindrical surface B, so that a very large storage space is required for storing omnidirectional image data. Required. Therefore, if you want to cover an image considerably above the horizontal plane, create omnidirectional image data by spherical projection as described above,
It would be advantageous to save it in a projected form.

Such omnidirectional image data includes, for example,
As described with reference to FIG. 2, four shooting locations slightly shifted are created and stored. Such 4
From the three omnidirectional image data, as described above, the omnidirectional image data for the left eye and the right eye corresponding to the azimuth of the line of sight is selected and shown to the left and right eyes of the observer, thereby realizing a stereoscopic view with a wide visual field. .

In the case of using the omnidirectional image data of the cylindrical projection, similarly to the case of using the omnidirectional image data of the spherical projection, the omnidirectional image data shown to the right eye or the left eye is not suddenly switched during the movement of the line of sight. Before and after the switch
The image data may be continuously switched while the two omnidirectional image data are overlapped and displayed. Further, the omnidirectional image data of the cylindrical surface projection can be displayed as it is, or the data projected on the display plane can be displayed.

An example of a stereo image display apparatus for displaying a stereo image with a wide field of view by the method of the present invention as described above will be described with reference to FIGS.

FIG. 3 is a schematic block diagram of a stereo image display device. In FIG. 3, reference numeral 100 denotes a storage device for omnidirectional image data, which stores, for example, four omnidirectional image data from a slightly shifted photographing point as described above.
Reference numeral 102 denotes a line-of-sight input device for inputting the direction of the line of sight of the observer. The line-of-sight input device 102 may be a device for inputting the line-of-sight direction by manual operation such as a keyboard, or a known line-of-sight detecting device for automatically detecting and inputting the line-of-sight direction of an observer by an optical method or the like. Good. An image processing apparatus 104 creates and outputs left-eye image data and right-eye image data. As this image processing device, a general computer may be used, or a dedicated device may be used. When a computer is used, a program for executing each step of the processing described later with reference to FIGS. 4 to 6 is loaded. It should be noted that a medium on which such a program is recorded is naturally included in the present invention. A display device 106 displays the left-eye image data and the right-eye image data in a stereoscopic manner. As the display device 106, a known image display device such as a polarization type or a head mounted type (goggle type) that can separately observe a left eye image for a left eye and a right eye image for a right eye can be used.

FIG. 4 is an overall processing flowchart of the stereo image display device. The overall processing flow is as follows. First, the gaze direction of the observer is input from the gaze direction input device 102 to the image processing device 104 (step 200). In the image processing device 104, the omnidirectional image for the left eye is selected from the four omnidirectional image data stored in the storage device 100 as described with reference to FIG. The data and the omnidirectional image data for the right eye are selected (step 202), the image data for the left eye is created and output from the omnidirectional image data for the left eye, and the image data for the right eye is created from the omnidirectional image data for the right eye. Is output (step 204). The left-eye image data and the right-eye image data are displayed on the display device 106.
Is displayed as a stereo image (step 20).
6). As described above, by continuously changing the line-of-sight direction, it is possible to stereoscopically view an image in all directions.

When the omnidirectional image data stored in the storage device 100 is the projection data, the processing shown in FIG. That is, from the selected omnidirectional image data for the right eye and the left eye (projected in a plan view), the visual field regions of the left eye and the right eye are respectively cut out (step 300), and each is reprojected onto a spherical surface (step 302). Do you want to output it as is (Step 3
06) Alternatively, if necessary, the image is projected on a display plane perpendicular to the line of sight (step 304), and then output (step 306). An observer may be able to instruct whether or not to perform projection on the display plane.

If the omnidirectional image data stored in the storage device 100 is omnidirectional image data projected on a cylindrical surface, the process shown in FIG. 6 is performed in step 204. That is, from the selected omnidirectional image data for the right eye and the left eye, the visual field regions of the left eye and the right eye are respectively cut out (step 400), and are output as they are (step 404), or if necessary, the visual line direction is further increased. Are projected on a display plane perpendicular to the above (step 402), and then output (step 404). An observer may be able to instruct whether or not to perform projection on the display plane.

As described above, a plurality of (here, for example, four) omnidirectional image data stored in the storage device 100 may be either spherical projection or cylindrical projection. Very similar to each other. Therefore, based on omnidirectional image data from a certain shooting point,
If the data is compressed by a known still image compression method, and the omnidirectional image data from the other photographing points take difference data from the reference omnidirectional image data, if it is appropriately converted and data compressed, An extremely high compression ratio can be expected, and storage and transmission can be performed efficiently. Such compressed data may be stored in the storage device 100.
In this case, in the image processing device 104, it is necessary to separately perform a process of restoring omnidirectional image data from a required shooting point from the compressed data.

[0025]

As is apparent from the above description, according to the present invention, it is possible to stereoscopically view a wide-field image such as a 360-degree panoramic image without causing a break in the field of view. Can be obtained.

[Brief description of the drawings]

FIG. 1 is a diagram for explaining creation and storage of omnidirectional image data.

FIG. 2 is a diagram for explaining omnidirectional image data at different photographing points and selection based on a line-of-sight direction;

FIG. 3 is a block diagram illustrating an example of a stereo image display device.

FIG. 4 is a flowchart illustrating an overall processing flow of the stereoscopic image display device.

FIG. 5 is a flowchart of a process for creating and outputting image data for the left eye and the right eye from omnidirectional image data of the orthographic projection.

FIG. 6 is a flowchart of a process for creating and outputting left-eye and right-eye image data from omnidirectional image data projected on a cylindrical surface.

[Explanation of symbols]

 REFERENCE SIGNS LIST 100 storage device for omnidirectional image data 102 gaze direction input device 104 image processing device 106 display device A spherical surface on which image is projected and synthesized B cylindrical surface on which image is projected and synthesized C horizontal plane D disk on which synthesized image is projected and projected

Claims (2)

[Claims] 1. An omnidirectional image data for a left eye and an omnidirectional image data for a right eye selected from three or more omnidirectional image data in which photographing points prepared in advance are shifted based on the direction of a line of sight of an observer. A stereoscopic image display method, wherein the left and right eyes of the observer are shown respectively. 2. A storage means for storing three or more omnidirectional image data in which photographing points are shifted, a gaze direction input means for inputting a gaze direction of an observer, and a gaze input by the gaze direction input means. Omnidirectional image data for the left eye and omnidirectional image data for the right eye are selected from the three or more omnidirectional image data stored in the storage means, and the left eye omnidirectional image data is selected from the selected omnidirectional image data. Processing means for creating and outputting image data for the right eye and image data for the right eye, and a display for displaying the image data for the left eye and the image data for the right eye output from the image processing means as images for the left and right eyes of the observer, respectively. Stereo image display device comprising: